the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
NN4CAST: An end-to-end deep learning application for seasonal climate forecasts
Abstract. Predicting climate variables at seasonal time scales is crucial for climate services. At these time scales, the most important driver of the atmospheric variability patterns is the ocean, which can trigger local perturbations that could affect the climate of remote areas through different teleconnection mechanisms. Dynamical models are not always effective in overcoming the challenges posed by complex climate processes and may show a low signal-to-noise ratio in the response of some phenomena (such as ENSO). Recently, statistical approaches, which focus on the relationships between predictor and predictand fields, have gained popularity due to advances in computing capabilities and the availability of extensive meteorological datasets. This is particularly evidenced in the development of short-range weather prediction data-driven models. However, seasonal prediction remains challenging, especially in regions where there are complex non-linear remote interactions of non-stationary and seasonal dependent signals from different sources of predictability of the Earth system. For this reason, we have developed a non-linear modelling tool, named as Neural Network foreCAST (NN4CAST). It is a deep learning model library which provides a versatile tool for operational prediction and non-linear statistical analysis, which can enhance seasonal forecast accuracy and understanding of the underlying dynamics. It is demonstrated how NN4CAST is able to provide accurate seasonal predictions in regions where the atmospheric response to the ocean is mostly linear (i.e., tropics) as well as in remote areas through atmospheric teleconnections, where the signal is highly non-linear (i.e., North Atlantic). It is in those remote regions where the dynamical models fail to capture the climate signal, and where the deep learning models may be more useful for a seasonal data-driven or hybrid prediction system. Therefore, the application of this model could provide accurate and reliable forecasts that can be useful for the different strategic societal sectors such as marine ecosystems, health or energy, which require predictions on these time scales.
- Preprint
(2102 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2024-2897', Anonymous Referee #1, 24 Oct 2024
This is an interesting application for building sub-seasonal models but I have several concerns that would be good to address before it is published.
Major Corrections
1. It is not clear to me what the focus of this paper. Is it to present this new framework and the model you have trained is just an example of an application that could be done with the new framework? Or is the idea to present this new model? If the former, do you have plans to extend this into short lead time weather forecasting? It seems to me that most of what you have developed e.g. hyperparameter tuning and XAI could be useful here. If the latter then I think it would be good to have a better description of the model.
2. As you mention in the introduction, sub-seasonal forecasting is very uncertain. I think for this framework to have a significant impact, it would need to be able to include a way to quantify uncertainty. For example, allowing multiple initial conditions, injections of Gaussian noise or the generation of ensembles.
3. It is unclear to me who provides the model? Are there example models provided in the repository? Or can the user prepare their own models and what format should they be in? Torch/tensorflow?
4. For the model you present, I am not convinced that cross-validation is appropriate across an annual timescale. Is the idea to make the model robust against climate change? We know that ERA5 is also worse pre-1979 because of the lack of satellite observations.
Minor
1. I think in the discussion around line 65, it would be good to mention Neural GCM as an example of an effective hybrid model.
2. I am not sure Lines 73-74 follow. You say the models are largely linear and then you say that this is important for non-linear relationships?
3. I think large parts of Section 2 could be removed. The basic theory of neural networks does not need to be included in a paper.
Citation: https://doi.org/10.5194/egusphere-2024-2897-RC1 -
AC2: 'Reply on RC1', Víctor Galván Fraile, 22 Nov 2024
Dear Reviewer,
Thank you very much for your thoughtful and constructive comments. Our responses to your review are provided in the attached PDF document.
Best regards,
Víctor Galván and Co-authors
-
RC2: 'Reply on AC2', Anonymous Referee #1, 04 Dec 2024
Dear authors,
Thank you for your responses. Please find below my responses:
1. I'm afraid I still fail to see why this framework is specific to S2S. I think it would actually make a stronger paper if you said it could be used for doing short range forecasting too.
3. If this is the case then I think there needs to be more details on the model used. By this I do not mean the neural network theory (of which I still think there is too much) but instead the architecture you used for the results you showed e.g. number of layers etc.
4. Please could you add some of this clarification to the paper to make it more understandable for the reader.
Citation: https://doi.org/10.5194/egusphere-2024-2897-RC2 -
AC4: 'Reply on RC2', Víctor Galván Fraile, 04 Mar 2025
Dear Reviewer,
We sincerely appreciate your thoughtful feedback and the time you dedicated to reviewing our work. Enclosed, you will find a detailed response to your comments in the attached PDF, along with a revised version of the manuscript where all modifications from the original submission have been highlighted.
Best regards,
Víctor Galván and Co-authors
-
AC4: 'Reply on RC2', Víctor Galván Fraile, 04 Mar 2025
-
RC2: 'Reply on AC2', Anonymous Referee #1, 04 Dec 2024
-
AC2: 'Reply on RC1', Víctor Galván Fraile, 22 Nov 2024
-
CEC1: 'Comment on egusphere-2024-2897: No compliance with the policy of the journal', Juan Antonio Añel, 29 Oct 2024
Dear authors,
Unfortunately, after checking your manuscript, it has come to our attention that it does not comply with our "Code and Data Policy".
https://www.geoscientific-model-development.net/policies/code_and_data_policy.htmlYou have archived your code on GitHub (https://github.com/Victorgf00/nn4cast/tree/main). However, GitHub is not a suitable repository for scientific publication. GitHub itself instructs authors to use other long-term archival and publishing alternatives, such as Zenodo. Therefore, please publish your code in one of the appropriate repositories and reply to this comment with the relevant information (link and a permanent identifier for it (e.g. DOI)) as soon as possible, as we can not accept manuscripts in Discussions that do not comply with our policy. Therefore, the current situation with your manuscript is irregular.
In this way, if you do not fix this problem, we will have to reject your manuscript for publication in our journal.
Also, you must include the modified 'Code and Data Availability' section in a potentially reviewed manuscript, the DOI of the code (and another DOI for the dataset if necessary).
Juan A. Añel
Geosci. Model Dev. Executive EditorCitation: https://doi.org/10.5194/egusphere-2024-2897-CEC1 -
AC1: 'Reply on CEC1', Víctor Galván Fraile, 30 Oct 2024
Dear Dr. Añel,
Thank you for your feedback and for bringing the "Code and Data Policy" requirements to our attention. We apologize for the oversight in our initial submission.
We have now archived our code in compliance with the journal's requirements by depositing it in Zenodo. Please find the DOI for the code below:
DOI: 10.5281/zenodo.14011998
We have also updated the 'Code and Data Availability' section in our manuscript accordingly. The revised manuscript with the modified section will be submitted promptly.
Thank you once again for the guidance, and please let us know if there is anything further needed to facilitate the review process.
Sincerely,
Víctor Galván (on behalf of the authors team)Citation: https://doi.org/10.5194/egusphere-2024-2897-AC1 -
CEC2: 'Reply on AC1', Juan Antonio Añel, 30 Oct 2024
Dear authors,
Many thanks for addressing this issue so quickly. We can consider now the current version of your manuscript in compliance with the code policy of our journal.
Juan A. Añel
Geosci. Model Dev. Executive Editor
Citation: https://doi.org/10.5194/egusphere-2024-2897-CEC2
-
CEC2: 'Reply on AC1', Juan Antonio Añel, 30 Oct 2024
-
AC1: 'Reply on CEC1', Víctor Galván Fraile, 30 Oct 2024
-
RC3: 'Comment on egusphere-2024-2897', Brian Henn, 19 Feb 2025
Review of NN4CAST: An end-to-end deep learning application for seasonal climate forecasts, submitted to GMD, February 2025.
Brian Henn, Allen Institute for Artificial Intelligence, climate modeling team, Seattle, WA
General comments
The authors describe a software package (“NN4CAST”) that allows for training neural networks (NNs) to predict climate model fields in a simple, single-input to single-output manner. The software also allows for some plotting and interpretability of the trained NN model and its predictions. The authors describe what neural networks are, and how they may fit into seasonal predictions of climate variables, at a fairly elementary level. They then present an example in which their package has been used to predict monthly-averaged SLP fields from monthly-averaged SST fields with a lead time of about two months, showing where the predictions have more and less skill and providing some examples of the ML interpretability and attribution.
NN4CAST could provide a nice educational example of using simple NNs to predict single fields as outputs of other single fields. This type of software would help students learn about basic ML and NN implementations, and allow them to understand when simple predicability can be leveraged via these tools. It is nicely implemented from a usability standpoint, with flexible configurations for data preprocessing and simple NN model training/hyperparameter tuning. But, for the reasons I note below, it is unlikely to provide state of the art predictability improvements in the field of seasonal forecasting.
The authors note that seasonal forecasting may be nonlinear and that conventional statistical techniques are primarily linear, as a justification for using neural networks for season forecasting. However, they state that they choose an example (Northern hemisphere wintertime ENSO response) that they state is mostly linear. Thus, it is likely that their results would be the same if they used an off-the-shelf linear approach (eg linear regression) that does not require NN training. In my opinion, there would be much stronger justification for the package if the authors could show that it produces improved results in a nonlinear case, beyond what a linear baseline approach could achieve.
Relatedly, most of the benefits of using NNs come from “deep” learning where datasets are large and have complex relationships between predictors and predicands. NN4CAST as applied here is not capable of tackling these kinds of problems: for models with many deep layers and large numbers of parameters because training will become impractically slow without GPU support. As such, NN4CAST as currently written is limited to “toy” examples, where it perhaps does not outperform linear regression. The software is also written in simplistic ways that will not allow it to be extended to other datasets and large-scale use cases. For example, it does not use typing, object-oriented organization, etc. that would make it easier to modify to work on other cases.
Overall, however, the paper is clearly written, the figures are mostly quite informative, and the examples of software code snippets are quite helpful. As I mentioned, this is a useful tool for teaching beginner users of ML how to apply it towards earth science data.
Specific comments
L35: “Multidecadal ocean variability and the Global Warming trend alters the global circulation and, thus, the way in which atmospheric teleconnections (i.e., Rossby waves) propagate, introducing non-stationarities in the system”. It is probably worth noting that non-stationarity likely is a bigger problem for statistical models than it is for dynamical ones/ESMs, which can generalize to unseen regimes via physical laws. As a result this is not a very good justification for statistical approaches like NNs, unlike the other items in this list which do suggest that statistical approaches will be useful.
L61-L74: The authors note growth of ML weather models here, but should be clearer that these types of models are making autoregressive forecasts of the evolution of the atmosphere (and in some cases the coupled earth-ocean system, eg, this paper: https://arxiv.org/abs/2409.16247) at sub-daily temporal frequency. Some of these ML models are capable of making seasonal forecasts in this way. This is a much different and harder problem than making predictions of static snapshots of monthly-mean variables, which do not capture the high-frequency temporal variability of the climate. In my opinion the authors need to make the distinction clearer between Graphcast, etc., and NN4CAST in that regard – they are not really comparable in terms of what they forecast.
L102: “2 Theoretical framework”: While the material in this section is useful for beginners to the field, it is mostly what would be covered in an elementary textbook on machine learning. For that reason, it is likely not necessary to cover in this article. The authors can assume that readers either know this information already or can read it in other sources.
Equation 3: This is not the formula for ACC. This is a formula for a correlation coefficient between the predictions and the targets. ACC, however, is more complex: It is the correlation between anomalies from the mean prediction vs. anomalies from the mean target. See for example: https://wattclarity.com.au/other-resources/glossary/other-resources-glossary-anomaly-correlation-coefficient/
Listing 3: There is a typo here - the output of the first line is “outputs_hold_out”, but the inputs to the second line include “outputs_cross_validation” that is not defined in this block.
Figure 2: Note that it is more typical to use stippling for areas where there **is** significant correlation, rather than areas where there is **not** significant correlation as Figures 2, 3, and 4 do. I would suggest using the more conventional approach.
L310: “In this case, the metrics evaluating the performance of the model, depicted in Fig. 3, are slightly different than before. Concretely, the ACC map is smoother and less noisy globally (Fig. 3a).” I think this is misleading. My guess is that the reason the correlation maps are less noisy is not due to the k-folds approach, but simply that the map in Figure 3a is computed over a 4x longer period than the map in Figure 2a. A more correct comparison of the effects of k-folds would be to use the same test period (2000-2019) both with and without k-folds.
Figure 5b - It is visually difficult for me to assess the skill of the model in 5b when the predictions are plotted in shading and the observations are plotted in contours. An easier way for reasons to assess the skill of the predictions would be to plot two maps (one of shaded predictions and one of shaded observations, or one of shaded predictions and one of shaded model error with the same colorbar) side-by-side.
Figure 6b: This seems to just repeat Figure 5b -- does it need to be shown again?
Technical corrections
Note: I attempted to download the zipfile of the datasets used from Zenodo, and found that I was not able to uncompress the zipfile (an error occurred). I am not sure if this data is correctly archived.
Citation: https://doi.org/10.5194/egusphere-2024-2897-RC3 -
AC3: 'Reply on RC3', Víctor Galván Fraile, 04 Mar 2025
Dear Reviewer,
Thank you for your valuable and insightful feedback. We appreciate the time and effort you put into reviewing our work. Please find our detailed responses to your comments in the attached PDF document, as well as the revised version of the manuscript with the changes from the original submission highlighted.
Best regards,
Víctor Galván and Co-authors.
-
AC3: 'Reply on RC3', Víctor Galván Fraile, 04 Mar 2025
Data sets
NN4CAST_manual Víctor Galván Fraile et al. https://doi.org/10.5281/zenodo.13768761
Model code and software
NN4CAST Víctor Galván Fraile https://github.com/Victorgf00/nn4cast/tree/main
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
450 | 125 | 196 | 771 | 10 | 9 |
- HTML: 450
- PDF: 125
- XML: 196
- Total: 771
- BibTeX: 10
- EndNote: 9
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1