the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 10 Apr 2024
Do data-driven models beat numerical models in forecasting weather extremes? A comparison of IFS HRES, Pangu-Weather and GraphCast
Abstract. The last few years have witnessed the emergence of data-driven weather forecast models able to compete and in some respects outperform physics-based numerical models. However, recent studies question the capability of data-driven models to provide reliable forecasts of extreme events. Here, we aim to evaluate this claim by comparing the performance of leading data-driven models in a semi-operational setting, focusing on the prediction of near-surface temperature and windspeed extremes globally. We find that data-driven models outperform ECMWF’s physics-based deterministic model in the average prediction of 10 m windspeed and 2 m temperature, and can also compete with the physics-based model in terms of extremes in most regions. However, the choice of best model depends strongly on region, type of extreme and sometimes even lead time. Thus, we conclude that data-driven models may already now be a useful complement to physics-based forecasts in those regions where they display superior tail performance, but that some challenges still need to be overcome before widespread operational implementation can take place.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(22823 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(22823 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-1042', Anonymous Referee #1, 22 Apr 2024
It's really good to see more papers like this evaluating the current generation of ML models in more detail. Thank you to the authors for their good work! More detailed comments and suggestions are attached.Â
- AC1: 'Reply on RC1', Leonardo Olivetti, 19 Jul 2024
-
RC2: 'Comment on egusphere-2024-1042', Anonymous Referee #2, 24 Jun 2024
In their study, Olivetti and Messori evaluate the efficacy of data-driven models in forecasting extreme events by comparing the performance of leading data-driven models within a semi-operational context, specifically targeting the prediction of near-surface temperature and windspeed extremes on a global scale. The authors demonstrate that data-driven models surpass the European Centre for Medium-Range Weather Forecasts (ECMWF)'s physics-based deterministic model in the average prediction of 10m windspeed and 2m temperature. Furthermore, these models can rival the physics-based model in forecasting extreme events in most regions. However, the optimal model choice is highly dependent on the region, type of extreme event, and occasionally the lead time. Consequently, the authors conclude that data-driven models could serve as valuable supplements to physics-based forecasts in regions where they exhibit superior performance in predicting extreme values. Nonetheless, certain challenges must be addressed before these models can be widely adopted in operational settings
The paper is well written and has the potential to be a relevant paper for both the weather prediction community and the machine learning community. The authors thoroughly compare temperature extremes and wind extremes. These comparisons are valuable since they compare the most up-to-date data-driven models with state-of-the-art physics-based models. My main criticism is that they do not explore potential reasons for their results. In my view the authors should provide potential avenues for 1) every one of the evaluation metrics they have calculated, and 2) the potential regional-model-dependencies that might explain results at a regional level. Up to now the authors have just calculated metrics and described their results however some educated guesses and further discussion of why they get the results they get would benefit this paper.Â
I suggest the authors to make an effort to go beyond merely reporting results, finding and discussing potential causes for their results. Due to this, I suggest a major revision.
Citation: https://doi.org/10.5194/egusphere-2024-1042-RC2 - AC2: 'Reply on RC2', Leonardo Olivetti, 19 Jul 2024
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-1042', Anonymous Referee #1, 22 Apr 2024
It's really good to see more papers like this evaluating the current generation of ML models in more detail. Thank you to the authors for their good work! More detailed comments and suggestions are attached.Â
- AC1: 'Reply on RC1', Leonardo Olivetti, 19 Jul 2024
-
RC2: 'Comment on egusphere-2024-1042', Anonymous Referee #2, 24 Jun 2024
In their study, Olivetti and Messori evaluate the efficacy of data-driven models in forecasting extreme events by comparing the performance of leading data-driven models within a semi-operational context, specifically targeting the prediction of near-surface temperature and windspeed extremes on a global scale. The authors demonstrate that data-driven models surpass the European Centre for Medium-Range Weather Forecasts (ECMWF)'s physics-based deterministic model in the average prediction of 10m windspeed and 2m temperature. Furthermore, these models can rival the physics-based model in forecasting extreme events in most regions. However, the optimal model choice is highly dependent on the region, type of extreme event, and occasionally the lead time. Consequently, the authors conclude that data-driven models could serve as valuable supplements to physics-based forecasts in regions where they exhibit superior performance in predicting extreme values. Nonetheless, certain challenges must be addressed before these models can be widely adopted in operational settings
The paper is well written and has the potential to be a relevant paper for both the weather prediction community and the machine learning community. The authors thoroughly compare temperature extremes and wind extremes. These comparisons are valuable since they compare the most up-to-date data-driven models with state-of-the-art physics-based models. My main criticism is that they do not explore potential reasons for their results. In my view the authors should provide potential avenues for 1) every one of the evaluation metrics they have calculated, and 2) the potential regional-model-dependencies that might explain results at a regional level. Up to now the authors have just calculated metrics and described their results however some educated guesses and further discussion of why they get the results they get would benefit this paper.Â
I suggest the authors to make an effort to go beyond merely reporting results, finding and discussing potential causes for their results. Due to this, I suggest a major revision.
Citation: https://doi.org/10.5194/egusphere-2024-1042-RC2 - AC2: 'Reply on RC2', Leonardo Olivetti, 19 Jul 2024
Peer review completion
Journal article(s) based on this preprint
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 952 | 495 | 45 | 1,492 | 34 | 26 |
- HTML: 952
- PDF: 495
- XML: 45
- Total: 1,492
- BibTeX: 34
- EndNote: 26
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Leonardo Olivetti
Gabriele Messori
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(22823 KB) - Metadata XML