the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Deep Temporal Convolutional Networks for F10.7 Radiation Flux Short-Term Forecasting
Luyao Wang
Xiaoxin Zhang
Guangshuai Peng
Zheng Li
Abstract. F10.7, the solar radiation flux at a wavelength of 10.7 cm (F10.7), is often used as an important parameter input in various space weather models and is also a key parameter for measuring the strength of solar activity levels. Therefore, it is valuable to study and forecast F10.7. In this paper, the temporal convolutional network (TCN) approach in deep learning is used to predict the daily value of F10.7. The F10.7 series from 1957 to 2019 are used, which the datasets from 1957 to 2008 are used for training and the datasets from 2009 to 2019 are used for testing. The results show that the TCN model of prediction F10.7 with a root mean square error (RMSE) from 5.03 to 5.44sfu and correlation coefficients (R) as high as 0.98 during solar cycle 24. The overall accuracy of the TCN forecasts is better than those of the widely used autoregressive (AR) models and the results of the US Space Weather Prediction Center (SWPC) forecasts especially for 2 and 3 days ahead. In addition ,the TCN model is slightly better than other neural network models like backward propagation network (BP) and long short-term memory network (LSTM) in terms of the solar radiation flux F10.7 forecast. The TCN model predicted F10.7 with a lower root mean square error, a higher correlation coefficient and the better overall model prediction.
- Preprint
(955 KB) - Metadata XML
-
Supplement
(225 KB) - BibTeX
- EndNote
Luyao Wang et al.
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2023-1801', Anonymous Referee #1, 06 Sep 2023
I have read the manuscript "Deep Temporal Convolutional Networks for F10.7 Radiation Flux Short-Term Forecasting" by Wang et al. The authors present a new approach to build a predictive F10.7 model which exhibits very promising results. Nevertheless, I have two major comments.
1) The language used in many cases is really bad and confusing to the reader. Please take careful care of the syntax and rewrite the manuscript where needed. I have also pointed out several cases in the attached pdf file.
2) The authors use approximately 4 solar cycles for the training of the ML scheme and 1 solar cycle (solar cycle 24) as a test dataset. Even though this is a pretty usual technique to validate a model, it can potenitally lead to significant misconceptions. This is due to the fact that the solar cycle 24 is quite weak compared to previous cycles (this is also something that it is not discussed in the text at all). A more robust technique would be to use an iterative leave-one-out method, which is described in detail in Aminalragia-Giamini et al. 2020 (https://doi.org/10.1051/swsc/2019043). A suggestion could be that the authors leave iteratively one solar cycle out as a test dataset and rerun the model each time (e.g. keep SC23 as test dataset and train the model with the rest SCs, then keep SC22 as test dataset and train the model with the rest SCs, etc.). In the end they can evaluate the metrics (MAE, RMSE, etc.) using the predictions of all solar cycles.
I have several minor and syntax comments as well, which are provided in the pdf file attached.
-
AC1: 'Reply on RC1', lu yao wang, 02 Oct 2023
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2023/egusphere-2023-1801/egusphere-2023-1801-AC1-supplement.zip
-
AC1: 'Reply on RC1', lu yao wang, 02 Oct 2023
-
RC2: 'Comment on egusphere-2023-1801', Anonymous Referee #2, 13 Sep 2023
General comments
In this paper, the authors describe and investigate a deep-learning-based model dedicated to forecasting the daily F10.7 index. The model is constructed using an architecture known as Temporal Convolutional Network (TCN), which employs specialized convolutional kernels designed for processing sequential data like time series. The authors provide forecasts for the F10.7 index ranging from 1 to 3 days ahead and evaluate their model using three common metrics: Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Pearson's correlation coefficient. They compare their model's performance to forecasts generated by the US Space Weather Prediction Center (SWPC) and another so-called auto-regressive (AR) model and observe that their new model significantly outperforms the SWPC and AR baselines. Additionally, they compare it to other deep learning-based models and conclude that their model represents an improvement over existing architectures, including Long Short-Term Memory (LSTM) networks. The key interest of the paper is the quality of these forecasts and their significant improvement over the state of the art, which would be worthy of publication.
In my opinion however, this manuscript is marred by a significant number of flaws, both in content and form, which prevent me from recommending it for publication unless it undergoes a thorough revision.
Indeed, the quality of the English used in this manuscript is at times rather weak, and many sentences are unclear or become ambiguous as a result. I understand that writing a manuscript in a foreign language can be arduous and require considerable extra effort. That's why I suggest that authors, if they have the opportunity, have their manuscript proofread by an English-speaking person who can help them improve its fluency. In particular, I recommend avoiding overuse of the passive voice, which can make reading confusing, and thoroughly checking the syntax.
More problematically, the authors occasionally employ exaggerated statements, that are inappropriate for a scientific manuscript and lacks supporting references. This is particularly evident in section 2.3, where they describe the methodology (please refer to specific comments below).
I also find several aspects regarding the architecture of the proposed model and the comparisons with other models somewhat unclear. Additionally, their assertive conclusions seem disproportionately strong given the limited scope of their comparisons, often relying on a single metric. In my view, there isn't sufficient evidence to substantiate the authors' conclusions yet and more comprehensive comparisons with the baseline models are needed.
My comments and recommendations are listed below.
Specific Comments
- L. 29-30: I disagree with the author’s definition of a time series. A time series does not necessarily have a fixed interval between points.
- L. 33-35: The authors enumerate several institutions that are presumed to be the primary sources for F10.7 forecasts. Instead, the authors could simply list and reference the forecast models, as they do shortly afterward. Alternatively, if they wish to mention institutions offering operational F10.7 forecasts, they should rephrase their sentence to eliminate any implication of ranking. Without a credible source, this ranking lacks scientific validity and holds no relevance in a scientific article.
- L. 58: The authors should provide a reference to back up their claims about RNNs (which, as far as I know, are correct).
- L. 59-60: I fail to see how the TCN's capacity for parallelizing calculations could be a result of reading data faster (in fact, the causal relationship might be the reverse). If the authors intend to highlight that TCN trains faster than an LSTM, they should consider rephrasing their sentence.
- Section 2.1: To complete their presentation of the F10.7 index, we suggest that the authors include a histogram of the distribution of possible F10.7 values, along with an autocorrelation plot. This would likely be very helpful for readers who are not very familiar with the study of this index.
- Section 2.1: The authors only use a train set and a test set, that is also used for validation. I strongly recommend that the authors use a split into three sub-sets: training, validation and testing. Indeed, when the same set is used for both validation and testing, it can lead authors to select an architecture and model hyperparameters that yield optimal results on this particular set. This approach is occasionally adopted when the dataset is too small to be divided into three adequately sized subsets, but this is not the situation here.
- Section 2.1: The authors use solar cycle 24 as their test case. This solar cycle is known as a very low-activity solar cycle. It would be more interesting to have test results on another cycle, such as cycle 23. To achieve this, the authors could implement cross-validation or, at the very least, train the model using a separate data split distinct from the initial one. It appears that the authors used the years 2003 and 2004 for testing as well, but it is unclear if they used the results from the training set, or if they trained a different model (see comment 24).
- Section 2.1: The authors mention they use processed data, but do not explain how they processed it. Please elaborate.
- Section 2.2: Surprisingly, this section consists of just one table. We suggest that the authors merge this section with sections 2.3 and/or 2.4, or put this table in the Appendix. On a side note, it's nice to see that this model can run on a "normal" PC and doesn't require a large compute server, which should make it easier to replicate.
- L. 80-81: This sentence is just a spectacular publicity stunt, without any scientific value, and is not supported by any source. It should be removed.
- L. 83-85: As per my understanding, the fact that TCNs do not suffer from information leakage from future time steps is what distinguishes it from classical 1D-CNNs and not from RNNs. Besides, it has nothing to due with RNNs “gate mechanisms”. Please comment and/or correct.
- Section 2.3: It would help if the authors could define what the vanishing gradient problem is, and explain a little more clearly how the TCN's particular architecture helps to avoid it.
- Section 2.3: I find the overall explanation of the TCN to be unclear. Specifically, there is ambiguity regarding the terms “causal convolution”, “inflated causal convolution” (Line 101), “causal extended convolution” (Line 106), and “dilated causal convolution” (Figure 4). This section should be revised to reduce confusion.
- L. 131: Define ReLU.
- L. 132: Are weights normalized with Batch Normalization? Please clarify.
- Table 2: I find this table very confusing.
- How is the batch size “None”?
- What unit is the step length?
- Clarify what input “dimension” and “shape” are.
- Why do “tcn_layer.receptive_field” and “Dense(1)” parameters not have a value?
- Section 2.5: Although MAE, RMSE, and R are indeed three commonly used measures, they may not offer a comprehensive assessment of a model's quality.
- I recommend that the authors consider following the guidelines provided by [1] and include additional bias and/or discrimination metrics, such as Probability of Detection and False Alarm Rate, to enhance the completeness of their evaluation.
- It would be even more beneficial if the authors could assess the timeliness of their forecasts using Dynamic Time Warping (DTW)-based metrics, as demonstrated by [2], [3].
- Section 3:
- It is very surprising that the model seems to perform sometimes better for 3-days ahead forecasts rather than 2-days or even 1-day ahead forecasts. Can the authors please comment?
- Since the authors indicated earlier that one of the advantages of a TCN is its training time, I suggest that the authors indicate the training time of their model.
- Figure 5:
- The Figure is too small to be useful. Any “persistence-like” (i.e. using the last true observation as the forecast) behavior would be hidden. Please resize and zoom-in, one or two month(s) of data is probably enough.
- In addition, sub-figure vertical axes should start at 0.
- Section 3 and Figure 6:
- The authors should describe and reference the model used by the SWPC to forecast the F10.7 index.
- They should also detail how they got these SWPC results (did they reproduce it? Did they download it?).
- It is also unclear why the comparison is limited to the years 2009 to 2013 and does not extend until 2019. Please provide an explanation for this choice and, if possible, complete the evaluation with the remaining years.
- Section 3 and Figure 7:
- it is unclear to me which model is the so-called “AR” model. Please clarify.
- It is again unclear why the model is evaluated only for those specific years. Please explain and if possible complete the evaluation with the remaining years.
- Section 3 and Table 4:
- The authors are now using the years 2003 and 2004 to evaluate and compare their model. Did they train another version of the model with a different train/test split? Please comment.
- Please comment why the RMSE is the only measure used for comparison purposes. In my opinion, the authors could additionally use at least one measure of correlation and one of bias (or even a measure of training time) in their comparison.
- Please clarify if all the BP and LSTM results were reproduced or obtained from their original authors. If the results were reproduced, it would be beneficial to have a description of the models’ architectures and training procedures (at least as an Appendix).
- I find it rushed to assert that the TCN architecture is intrinsically better than an LSTM for predicting the daily value of F10.7 when only one evaluation metric is used, over only two test years (even if these are years of high activity), without knowing anything about how the LSTM was trained (same sequence length?). In my opinion, the authors should comment on these points, and probably be more careful when asserting that TCN is a significant improvement over an LSTM. Additional metrics, or even figures to back up their claim, would be more convincing.
- Conclusion: The conclusion should probably be reworked and tempered after the above points have been addressed.
Technical Comments
- Many typing errors, e.g. on lines: 22; 25; 37; 42; 48; 50; 53; 166; 175; 176; 183; 188; 191; 195; 204; 268; etc.
- The authors should indicate in their abstract what is the forecast horizon associated with the provided performance metrics.
- The authors should specify in the abstract that their model is an autoregressive model (it only takes past values of the F10.7 index as inputs).
- The abstract is written in a rather poor and confusing language (in particular Lines 11 – 15), and should be rewritten.
- L. 30: Did the authors meant “correlation” instead of “link”? This sentence should be rephrased.
- L. 45-46: This sentence is very poorly written and should be completely reworded.
- In the manuscript, the authors often refer to a so-called Back-Propagation (BP) network. I understand that this denomination comes from the article by Xiao et al., 2017 cited by the authors. I find it a misnomer because "back-propagation" is the method used to make the neural network learn, and does not depend on the architecture of the network. For example, the vast majority of convolutional networks also use "back-propagation" during their training phrase (and the author’s TCN model also does). Here, the network referred to by the authors is simply a “feedforward artificial neural network” (also sometimes called “multi-layer perceptron”). This is why I suggest that the authors change the BP and BPNN denomination to a more standard and understandable one.
- L. 55-56: This sentence is very poorly written and should be reworded. “sfu” should be introduced.
- L. 56: The authors mention “RNN-based” architectures without introducing the meaning of the acronym and explaining that “RNN-based” methods include “LSTMs”.
- L. 58: The right reference is probably Bai et al., 2018 and not 2017.
- Section 2.3: Notation for vector is inconsistent. Sometimes the vector "x" is referred as "x" sometimes as 𝑥⃑ (with an upper arrow).
- L. 124: Please add a reference for L1 loss.
- L. 138-139: “Business sector” is probably a confusing way of referring to operational space weather centers or space agencies. Please clarify.
- Figure 5: “Practical” is unclear. Please consider changing it to “Observations” or something similar.
References
[1] M. W. Liemohn, A. D. Shane, A. R. Azari, A. K. Petersen, B. M. Swiger, and A. Mukhopadhyay, “RMSE is not enough: Guidelines to robust data-model comparisons for magnetospheric physics,” J. Atmospheric Sol.-Terr. Phys., vol. 218, p. 105624, Jul. 2021, doi: 10.1016/j.jastp.2021.105624.
[2] E. Samara et al., “Dynamic Time Warping as a Means of Assessing Solar Wind Time Series,” Astrophys. J., vol. 927, no. 2, p. 187, Mar. 2022, doi: 10.3847/1538-4357/ac4af6.
[3] G. Bernoux, A. Brunet, É. Buchlin, M. Janvier, and A. Sicard, “An operational approach to forecast the Earth’s radiation belts dynamics,” J. Space Weather Space Clim., vol. 11, p. 60, 2021, doi: 10.1051/swsc/2021045.
Citation: https://doi.org/10.5194/egusphere-2023-1801-RC2 -
AC2: 'Reply on RC2', lu yao wang, 02 Oct 2023
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2023/egusphere-2023-1801/egusphere-2023-1801-AC2-supplement.zip
Luyao Wang et al.
Luyao Wang et al.
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 105 | 53 | 12 | 170 | 17 | 6 | 6 |
- HTML: 105
- PDF: 53
- XML: 12
- Total: 170
- Supplement: 17
- BibTeX: 6
- EndNote: 6
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1