the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 27 Nov 2024
Improving the fine structure of intense rainfall forecast by a designed adversarial generation network
Abstract. Accurate short-term precipitation forecasting is critical for socio-economic activities. However, due to inherent deficiencies of numerical weather models, the accuracy of precipitation forecasts remains significantly inadequate. In recent years, deep learning has been employed to enhance precipitation forecasts, yet these forecasts frequently appear blurry and fail to meet the precision required for operational applications. In this paper, we propose a Generative Adversarial Fusion Network (GFRNet) designed to provide quantitative forecasts of 3-hour accumulated precipitation over the next 24 hours in North China, based on the outputs of multiple numerical weather models. Evaluation results indicate that GFRNet outperforms numerical models across all precipitation intensities. Specifically, GFRNet's threat scores (TS) improved by 4 %, 28 %, 35 %, and 19 % at thresholds of 0.1 mm, 10 mm, 20 mm, and 40 mm, respectively, compared to the highest spatial resolution regional numerical model of the China Meteorological Administration (CMA-3KM). Additionally, GFRNet's Fraction Skill Scores (FSS) at thresholds of 10 mm, 20 mm, and 40 mm show improvements of 13 %, 18 %, and 15 % respectively, over those of CMA-3KM. These enhancements are consistent across most spatial regions and forecast lead times. Furthermore, GFRNet outperforms all models in terms of Root Mean Square Error (RMSE) and Multi-Scale Structural Similarity Index (MS-SSIM). Compared to the deep learning-based precipitation model FRNet, which lacks a generative strategy and tends to produce blurry forecasts with over-prediction, GFRNet more accurately captures the fine structure and evolutionary patterns of precipitation, demonstrating significant operational value.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(5847 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(5847 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
CEC1: 'Comment on egusphere-2024-2888', Juan Antonio Añel, 27 Dec 2024
-
AC1: 'Code pipeline refinement', Zuliang Fang, 15 Jan 2025
Dear Editor,
Thank you very much for bringing this issue to my attention. Based on your valuable feedback and the journal's code requirements, I have reviewed and standardized the code format and supplemented the necessary information. You can find the updated materials at https://zenodo.org/records/14652556.If there are any further omissions or areas for improvement, please do not hesitate to let me know. I truly appreciate your guidance and will continue to make any necessary adjustments.
Best regards
Citation: https://doi.org/10.5194/egusphere-2024-2888-AC1
-
AC1: 'Code pipeline refinement', Zuliang Fang, 15 Jan 2025
-
RC1: 'A review of "Improving the fine structure of intense rainfall forecast by a designed adversarial generation network"', Anonymous Referee #1, 08 Jan 2025
The paper presents an application of deep learning techniques, specifically U-Net and GAN-based models, to enhance short-term precipitation forecasting, with a focus on the fine-scale structure of intense rainfall events. The authors compare the accuracy of three numerical weather prediction (NWP) models against two deep learning techniques: a U-Net-based model (FRNet) and a GAN-based model (GFRNet). They use multiple evaluation metrics to assess the relative performances of these approaches. The goal is to evaluate the accuracy of predicting 3-hourly accumulated precipitation over the next 24 hours for a region in North China. The paper shows many metrics and concludes that GFRNet demonstrates significant operational value.The paper deals with an interesting topic, related to improving the forecasting of the fine structure of intense rainfall. Moreover, it makes use of some current tools in deep learning, which seem promising for future operational use. However, I have two main concerns about the paper. The first one is the rationale of the model itself, as I do not understand how the fine structure of rainfall may be solely explained with the additional information (on top of the NWPs) provided to the GAN. The second one is related to the experimental design and the fairness of the comparisons and analyses provided. In my opinion, the paper requires an improvement of the rationale and the experimental design, as well as additional analyses before being considered for publication in a scientific journal.Below, I provide more details about the issues that I observe with the work.My main concern with the rationale of the paper is that it is not immediately evident how the information about Elevation, Latitude, Longitude, Cycle, and Lead Hour contributes independently to improving forecasts when much of this information may already be embedded in the NWPs. A mechanism should be presented or outlined to justify the gains in accuracy. If none exists, then all the required information about the fine structure of precipitation is already included in the original NWPs, and thus the methods presented are just extracting this information.If this is the case, as I believe (although I may be wrong), there may be other alternative methods that could improve forecasting with reduced complexity. To verify this point, I suggest that the authors include additional models, such as SVMs or Random Forests, to test if simpler bagging methods with far fewer parameters could also improve forecasting accuracy. In my experience, basic machine learning methods tend to perform similarly to deep learning ones (in this kind of application) at a significantly reduced level of complexity. Including these additional comparisons may serve to justify whether a GAN is an optimal strategy and to show if the improvement in forecasting accuracy comes from the deep learning techniques or from the combination of different sources of information.Using FRNet as a benchmark may not be entirely fair, as GFRNet is essentially an enhanced version of the same model with a more advanced training procedure. It would also be interesting to report training and inference times for all models used (I know that the authors have included part of this information in their manuscript).Thus, with respect to my concern with the rationale of the paper, the authors should include additional simpler models to check if GANs are justified for their complexity level or if other simpler methods may work similarly. Additionally, they should provide some insight into why the improvement occurs. This point leads to my concerns with the experimental design.My first concern with the experimental design is related to the limited amount of data used in the study. Using only four years of data may not be sufficient to fully verify the accuracy and robustness of the forecasting method. I understand that data limitations are difficult to overcome, but given the complexity of the models used, it is difficult to ensure that overfitting is not playing a role in the analysis. This concern is exacerbated by the fact that the original time series must be split into training, validation, and test sets.Moreover, the initial data selection may bias the results. NWPs provide continuous forecasts, so alternative methods should ideally also deliver continuous predictions to ensure a fair comparison. If a heavy data selection procedure is implemented, the comparison may not be entirely fair to the NWPs.A second concern with the experimental design is that I would have appreciated a clearer discussion in the methods' section about how the evaluation of accuracy was performed. Table 3 presents evaluation metrics and highlights the best and second-best performers. However, more attention should be paid to the differences. Are they significantly different? Or could all the methods (NWPs and nets) perform similarly given the amount of information used? What would be the expected distribution of the accuracy metrics? I am not fully convinced that part of the results are not an analysis of statistical fluctuations. Additionally, Table 4 seems to contradict the abstract, which states that "GFRNet outperforms all models in terms of Root Mean Square Error (RMSE)," but I may have missed something.A third concern is related to the case studies, which I believe should be justified and presented in a more detailed way. This point may be related to my concern about the rationale of the paper: if a mechanism by which the GAN strategy improves the forecast is provided, then the case studies may focus on clear examples of this mechanism at play. Without this, I believe a general statistical evaluation would provide a clearer representation of the model's advantages. A detailed analysis of specific situations may not be as illuminating.A fourth concern about the experimental design is related to the selection of three NWPs. If two models similar to ECMWF were available, would it make sense to include both? How would the results change? This raises questions about generalizability. In many machine learning applications, the data exert a closer control on accuracy than the algorithms themselves. I understand that a paper cannot address every concern, but some guidance from the authors would be appreciated.My final concern relates to the generality of the conclusions and the reproducibility of the results in other locations. How robust are the results to the data selection procedure or the structure of the ANN? Would the same structure work well in other locations, or would changes be required? If a less intense data selection procedure were used, how would the results change? If 20 years of data were available, would GFRNet perform similarly? I believe the study would be much more robust if extended to other regions with more data available. Currently, the method seems to work, but the evidence may not yet be robust enough to fully support the claims made in the paper.Finally, I present some comments about minor issues:+ References should be enclosed in parentheses. The way they are written now complicates the reading of the paper.+ Figures 4 and 5 are difficult to interpret, particularly due to their complex visual layout. A more intuitive representation could enhance their clarity. For the maps, since topography seems to play such an important role, residuals might provide better insights.+ Some references to equations are incomplete.+ A better discussion on ensemble forecasts and deterministic quantitative forecasts may be in order. In my opinion, ensemble forecasts may convey a much better idea of severe storm potential, especially when combined with synthetic generation, so focusing on deterministic forecasts may be a disadvantage.Citation: https://doi.org/
10.5194/egusphere-2024-2888-RC1 -
AC2: 'Reply on RC1', Zuliang Fang, 17 Jan 2025
Thank you very much for your thorough review and insightful comments and suggestions. Based on your feedback, I have identified and categorized nine specific issues, which I have addressed in detail and with focus. Please see the attached document for further information, where your questions are italicized, and my corresponding responses are provided in regular font.
For the sections of the manuscript that require modifications based on your suggestions, I have already made the necessary revisions offline. These updates will be incorporated into the manuscript after the preprint public review period ends. I sincerely hope you will continue to provide your valuable comments and guidance during this process.
Thank you again for your time and effort in reviewing my work.
Best regards
-
AC2: 'Reply on RC1', Zuliang Fang, 17 Jan 2025
-
RC2: 'Comment on egusphere-2024-2888', Anonymous Referee #2, 03 Apr 2025
Please refer to the PDF document for comments.
- AC3: 'Reply on RC2', Zuliang Fang, 08 Apr 2025
Interactive discussion
Status: closed
-
CEC1: 'Comment on egusphere-2024-2888', Juan Antonio Añel, 27 Dec 2024
Dear authors,
I would like to point you out an issue regarding the code you have shared as part of your submitted work. I have seen in some pieces of code paths that point out to local systems, such as "/THL8/home/zhq/fzl/branch/forecast" that are not available in the assets that you have shared. I would kindly request you to double-check your code and fix these issues where they are relevant, paying special attention to share all the assets that are necessary to replicate your work (scripts, models, input and output data).
Thanks,
Juan A. Añel
Geosci. Model Dev. Executive Editor
Citation: https://doi.org/10.5194/egusphere-2024-2888-CEC1 -
AC1: 'Code pipeline refinement', Zuliang Fang, 15 Jan 2025
Dear Editor,
Thank you very much for bringing this issue to my attention. Based on your valuable feedback and the journal's code requirements, I have reviewed and standardized the code format and supplemented the necessary information. You can find the updated materials at https://zenodo.org/records/14652556.If there are any further omissions or areas for improvement, please do not hesitate to let me know. I truly appreciate your guidance and will continue to make any necessary adjustments.
Best regards
Citation: https://doi.org/10.5194/egusphere-2024-2888-AC1
-
AC1: 'Code pipeline refinement', Zuliang Fang, 15 Jan 2025
-
RC1: 'A review of "Improving the fine structure of intense rainfall forecast by a designed adversarial generation network"', Anonymous Referee #1, 08 Jan 2025
The paper presents an application of deep learning techniques, specifically U-Net and GAN-based models, to enhance short-term precipitation forecasting, with a focus on the fine-scale structure of intense rainfall events. The authors compare the accuracy of three numerical weather prediction (NWP) models against two deep learning techniques: a U-Net-based model (FRNet) and a GAN-based model (GFRNet). They use multiple evaluation metrics to assess the relative performances of these approaches. The goal is to evaluate the accuracy of predicting 3-hourly accumulated precipitation over the next 24 hours for a region in North China. The paper shows many metrics and concludes that GFRNet demonstrates significant operational value.The paper deals with an interesting topic, related to improving the forecasting of the fine structure of intense rainfall. Moreover, it makes use of some current tools in deep learning, which seem promising for future operational use. However, I have two main concerns about the paper. The first one is the rationale of the model itself, as I do not understand how the fine structure of rainfall may be solely explained with the additional information (on top of the NWPs) provided to the GAN. The second one is related to the experimental design and the fairness of the comparisons and analyses provided. In my opinion, the paper requires an improvement of the rationale and the experimental design, as well as additional analyses before being considered for publication in a scientific journal.Below, I provide more details about the issues that I observe with the work.My main concern with the rationale of the paper is that it is not immediately evident how the information about Elevation, Latitude, Longitude, Cycle, and Lead Hour contributes independently to improving forecasts when much of this information may already be embedded in the NWPs. A mechanism should be presented or outlined to justify the gains in accuracy. If none exists, then all the required information about the fine structure of precipitation is already included in the original NWPs, and thus the methods presented are just extracting this information.If this is the case, as I believe (although I may be wrong), there may be other alternative methods that could improve forecasting with reduced complexity. To verify this point, I suggest that the authors include additional models, such as SVMs or Random Forests, to test if simpler bagging methods with far fewer parameters could also improve forecasting accuracy. In my experience, basic machine learning methods tend to perform similarly to deep learning ones (in this kind of application) at a significantly reduced level of complexity. Including these additional comparisons may serve to justify whether a GAN is an optimal strategy and to show if the improvement in forecasting accuracy comes from the deep learning techniques or from the combination of different sources of information.Using FRNet as a benchmark may not be entirely fair, as GFRNet is essentially an enhanced version of the same model with a more advanced training procedure. It would also be interesting to report training and inference times for all models used (I know that the authors have included part of this information in their manuscript).Thus, with respect to my concern with the rationale of the paper, the authors should include additional simpler models to check if GANs are justified for their complexity level or if other simpler methods may work similarly. Additionally, they should provide some insight into why the improvement occurs. This point leads to my concerns with the experimental design.My first concern with the experimental design is related to the limited amount of data used in the study. Using only four years of data may not be sufficient to fully verify the accuracy and robustness of the forecasting method. I understand that data limitations are difficult to overcome, but given the complexity of the models used, it is difficult to ensure that overfitting is not playing a role in the analysis. This concern is exacerbated by the fact that the original time series must be split into training, validation, and test sets.Moreover, the initial data selection may bias the results. NWPs provide continuous forecasts, so alternative methods should ideally also deliver continuous predictions to ensure a fair comparison. If a heavy data selection procedure is implemented, the comparison may not be entirely fair to the NWPs.A second concern with the experimental design is that I would have appreciated a clearer discussion in the methods' section about how the evaluation of accuracy was performed. Table 3 presents evaluation metrics and highlights the best and second-best performers. However, more attention should be paid to the differences. Are they significantly different? Or could all the methods (NWPs and nets) perform similarly given the amount of information used? What would be the expected distribution of the accuracy metrics? I am not fully convinced that part of the results are not an analysis of statistical fluctuations. Additionally, Table 4 seems to contradict the abstract, which states that "GFRNet outperforms all models in terms of Root Mean Square Error (RMSE)," but I may have missed something.A third concern is related to the case studies, which I believe should be justified and presented in a more detailed way. This point may be related to my concern about the rationale of the paper: if a mechanism by which the GAN strategy improves the forecast is provided, then the case studies may focus on clear examples of this mechanism at play. Without this, I believe a general statistical evaluation would provide a clearer representation of the model's advantages. A detailed analysis of specific situations may not be as illuminating.A fourth concern about the experimental design is related to the selection of three NWPs. If two models similar to ECMWF were available, would it make sense to include both? How would the results change? This raises questions about generalizability. In many machine learning applications, the data exert a closer control on accuracy than the algorithms themselves. I understand that a paper cannot address every concern, but some guidance from the authors would be appreciated.My final concern relates to the generality of the conclusions and the reproducibility of the results in other locations. How robust are the results to the data selection procedure or the structure of the ANN? Would the same structure work well in other locations, or would changes be required? If a less intense data selection procedure were used, how would the results change? If 20 years of data were available, would GFRNet perform similarly? I believe the study would be much more robust if extended to other regions with more data available. Currently, the method seems to work, but the evidence may not yet be robust enough to fully support the claims made in the paper.Finally, I present some comments about minor issues:+ References should be enclosed in parentheses. The way they are written now complicates the reading of the paper.+ Figures 4 and 5 are difficult to interpret, particularly due to their complex visual layout. A more intuitive representation could enhance their clarity. For the maps, since topography seems to play such an important role, residuals might provide better insights.+ Some references to equations are incomplete.+ A better discussion on ensemble forecasts and deterministic quantitative forecasts may be in order. In my opinion, ensemble forecasts may convey a much better idea of severe storm potential, especially when combined with synthetic generation, so focusing on deterministic forecasts may be a disadvantage.Citation: https://doi.org/
10.5194/egusphere-2024-2888-RC1 -
AC2: 'Reply on RC1', Zuliang Fang, 17 Jan 2025
Thank you very much for your thorough review and insightful comments and suggestions. Based on your feedback, I have identified and categorized nine specific issues, which I have addressed in detail and with focus. Please see the attached document for further information, where your questions are italicized, and my corresponding responses are provided in regular font.
For the sections of the manuscript that require modifications based on your suggestions, I have already made the necessary revisions offline. These updates will be incorporated into the manuscript after the preprint public review period ends. I sincerely hope you will continue to provide your valuable comments and guidance during this process.
Thank you again for your time and effort in reviewing my work.
Best regards
-
AC2: 'Reply on RC1', Zuliang Fang, 17 Jan 2025
-
RC2: 'Comment on egusphere-2024-2888', Anonymous Referee #2, 03 Apr 2025
Please refer to the PDF document for comments.
- AC3: 'Reply on RC2', Zuliang Fang, 08 Apr 2025
Peer review completion
Journal article(s) based on this preprint
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 775 | 162 | 27 | 964 | 18 | 29 |
- HTML: 775
- PDF: 162
- XML: 27
- Total: 964
- BibTeX: 18
- EndNote: 29
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Zuliang Fang
Qi Zhong
Haoming Chen
Xiuming Wang
Zhicha Zhang
Hongli Liang
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(5847 KB) - Metadata XML
Dear authors,
I would like to point you out an issue regarding the code you have shared as part of your submitted work. I have seen in some pieces of code paths that point out to local systems, such as "/THL8/home/zhq/fzl/branch/forecast" that are not available in the assets that you have shared. I would kindly request you to double-check your code and fix these issues where they are relevant, paying special attention to share all the assets that are necessary to replicate your work (scripts, models, input and output data).
Thanks,
Juan A. Añel
Geosci. Model Dev. Executive Editor