the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Infilling of Missing Rainfall Radar Data with a Memory-Assisted Deep Learning Approach
Abstract. Incomplete spatio-temporal meteorological observations can result in misinterpretations of the current climate state, uncertainties in early warning systems, or inaccuracies in nowcasting models and can thereby pose signficant challanges in hydrology research or similar applications. Traditional statistical methods for infilling missing precipitation data demand substantial computational resources and fail over large areas with sparse data – like temporary outages of weather radars. Although recent machine learning advancements have shown promise in addressing missing meteorological or satellite observations, they typically focus on spatial aspects, overlooking the complex spatio-temporal variability characteristic of precipitation, especially during extreme events. We propose a deep convolutional neural network enhanced with a temporal memory component to better account for temporal changes in precipitation fields. This approach can analyse arbitrary sequences from before and/or after the incomplete observation of interest. Our model is trained and evaluated on the hourly RADKLIM dataset, which features 1-km resolution precipitation derived from combined radar and weather station data across Germany. By infilling both synthetic and actual data gaps of RADKLIM, the study demonstrates the model's effectiveness, providing detailed insights into its capabilities during significant rainfall events, such as those in May 2012 and July 2021, including those responsible for the Ahrtal flood. This novel approach represents a step forward in hydrological applications, potentially improving the way we predict and manage water-related events by increasing the accuracy and reliability of precipitation data analysis.
- Preprint
(10857 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2024-1392', Anonymous Referee #1, 17 Dec 2024
The study of Meuer et al entitled “Infilling of Missing Rainfall Radar Data with a Memory-Assisted Deep Learning Approach” explores the use of machine learning for infilling of the RADOLAN precipitation product of the German Weather Service. The approach comprises a comparison with baseline and channel-based results. Although the proposed method is technically more demanding, it results in increased accuracy and in filled data gaps respecting the temporal and spatial pattern of the precipitation event.
The manuscript is written in a concise way, is well-structured and makes use of good English. However, to my impression, the actual discussion is missing and only few remarks could be interpreted as discussion. Therefore, please enlarge the discussion part and make use of more references to guide the interpretation. In this context, a broader discussion on the uncertainty of each infilling technique would also help to improve this section. Why were the events of May 2012 and July 2021 presented only?
To conclude, the manuscript requires further improvements my opinion before recommending it for publication in this journal.
Specific comments:
- 1, l. 2: “challenges”
p.2, l. 24: Please correct the figure number
- 3, l. 63: “depending” instead of “pending”
- 3, l. 70: Please add the specific product name (RADOLAN YW and RADOLAN RW)
- 4, l. 90: Please introduce the abbreviation
- 4, l. 95: this fact deserves a reference
- 5, l. 116: this reference is given twice, please remove one
- 9, l.189 – p.10, l. 191: please specify the hardware resources you used and give advice on some minimal requirement
- 10, l. 193: this may be the true final step, while “final” was already used on p. 7, l. 174. Please correct this.
Figure 1: Please add to the caption the data source and reference to the product.
Figure 7. The caption is the same as the one of Figure E1 although the maps are different. Please check the captions.
Citation: https://doi.org/10.5194/egusphere-2024-1392-RC1 -
AC2: 'Reply on RC1', Johannes Meuer, 19 Feb 2025
We thank reviewer #1 for the comments. In the following, the reviewer's comments are in plain text and our responses in bold text.
The study of Meuer et al entitled “Infilling of Missing Rainfall Radar Data with a Memory-Assisted Deep Learning Approach” explores the use of machine learning for infilling of the RADOLAN precipitation product of the German Weather Service. The approach comprises a comparison with baseline and channel-based results. Although the proposed method is technically more demanding, it results in increased accuracy and in filled data gaps respecting the temporal and spatial pattern of the precipitation event.
The manuscript is written in a concise way, is well-structured and makes use of good English. However, to my impression, the actual discussion is missing and only few remarks could be interpreted as discussion. Therefore, please enlarge the discussion part and make use of more references to guide the interpretation. In this context, a broader discussion on the uncertainty of each infilling technique would also help to improve this section. Why were the events of May 2012 and July 2021 presented only?
Response: We thank the reviewer for the thorough review and for these kind remarks. We agree to extend the discussion section, and to include a treatment of each of the infilling techniques. In addition, we would like to explain that the May 2012 and July 2021 events were chosen because they represent an “average” rainfall event (2012) and a very extreme rainfall event (2021), and thereby provide a good illustration of the capabilities of the algorithms. In addition, we provide information on other periods, as shown in Figures 3 and 4, for the years 2007, 2012 and 2016.
To conclude, the manuscript requires further improvements my opinion before recommending it for publication in this journal.
Response: we thank the reviewer for making the points above, and the editorial comment below, which we will include in the revision of our paper.
Specific comments:
p.1, l. 2: “challenges”
Response: Thanks, we will correct this.
p.2, l. 24: Please correct the figure number
Response: We will adjust to “Figure 1”.
p.3, l. 64: “depending” instead of “pending”
Response: We will correct this.
p.3, l. 70: Please add the specific product name (RADOLAN YW and RADOLAN RW)
Response: We added a sentence, stating that we are using the RW product.
p.4, l. 90: Please introduce the abbreviation
Response: we will add “Convolutional Neural Network (CNN)”.
p.4, l. 95: this fact deserves a reference
Response: We added two references that deal with spatio-temporal nowcasting of precipitation (Shi et al. 2015, Tian et al. 2019).
p.5, l. 116: this reference is given twice, please remove one
Response: we will correct this.
p.9, l.189 – p.10, l. 191: please specify the hardware resources you used and give advice on some minimal requirement
Response: We added a “Technical Implementation” subsection in the methodology section where we specify the hardware resources and a framework description. It is, however, difficult to give some minimal requirements. In theory, this model could be run on any laptop after training.
p.10, l. 193: this may be the true final step, while “final” was already used on p. 7, l. 174. Please correct this.
Response: We changed this to “In another experiment …”.
Figure 1: Please add to the caption the data source and reference to the product.
Response: we will add the reference to DWD RADKLIM here.
Figure 7. The caption is the same as the one of Figure E1 although the maps are different. Please check the captions.
Response: we will adjust the caption of Figure E1, to indicate that this is the reconstruction for the entire area of Germany; and Fig. 7 is only a zoomed in portion. Additionally, in the caption of Fig. 7, we will indicate that Fig. E1 includes the entire territory of Germany.
Citation: https://doi.org/10.5194/egusphere-2024-1392-AC2
-
RC2: 'Comment on egusphere-2024-1392', Anonymous Referee #2, 09 Jan 2025
## General comments
The authors propose and analyze a new approach to the interpolation of values for incomplete precipitation maps. The authors motivate their study by explaining that such interpolation is often necessary as result of various factors, such as radar outages. They explain the structure of their “channel-based” approach, combined with an LSTM module, and apply it to a dataset provided by Germany’s national meteorological service. Finally, they show via a range of metrics and modeling scenarios that their proposed approach may outperform more basic ones.
The overall quality of the writing, referencing, structure and experiments is good. There do not appear to be any notable and relevant works missing from the list of references. Although not providing ground-breaking contributions to the field, the results are nonetheless valuable in illustrating the feasibility of alternative approaches to the problem of precipitation map in-painting using temporal data. The description and review of the addressed problem is also good. The manuscript would be suitable for publication in the journal after minor revisions to address the following concerns and questions. Particularly notable insightful and clear parts of the manuscript are also noted below.
## Review ratings
Scientific significance: Good
Scientific quality: Good
Presentation quality: Fair
## Specific and technical comments
Line 24: Unclear if this is in fact referring to Figure 1.
Line 40: More detail should be provided to explain what is meant by “data-driven”. I assume such methods are machine learning adjacent ones which contrast to statistical methods driven by modeling assumptions, but additional clarity would be helpful here.
Line 47: Additional detail would be beneficial here. It appears unfair to state that data-driven methods only account for spatial relationships while statistical methods only account for temporal variability. Explanation of the caveats of this statement should be included. For example, why is spatial interpolation, a statistical method, unable to reconstruct spatial features?
Line 59: The explanation of the dataset is clear and well organized. However, it is unclear which exact time period is used for this study.
Line 85: Please provide a citation for this statement.
Equations 1: Please use \mathrm to format the text
Line 102: This paper seems to be a seminal work in the field, and the overall approach of the presented work appears rather similar. Since many readers would already have this particular paper in mind when reading through this manuscript, it would be ideal to mention this paper earlier in the introduction and explain its similarities and differences with this work.
Line 122: There should be a brief mention of the programming language and packages used to implement the model, hardware utilized, and run time here.
Line 125: The experiments and associated results in this section are highly yet appropriately detailed. However, it is difficult at times to follow what is going on. Some reorganization here could be beneficial to the reader’s understanding. For example, the results and discussion could be split into at least two different sections.
Line 137: Please use \mathrm to format the text
Figure 2: This is a good way to present the relevant metrics in a visually appealing and informative manner
Figure 7: The text in this figure is difficult to read as it is too small.
Line 198: The conclusion is clear and provides a high-level summary of the main findings. A very brief reiteration of the motivation for the study here would also be helpful.
Citation: https://doi.org/10.5194/egusphere-2024-1392-RC2 -
AC1: 'Reply on RC2', Johannes Meuer, 19 Feb 2025
We thank reviewer #2 for the comments. In the following, the reviewer's comments are in plain text and our responses in bold text.
## General comments
The authors propose and analyze a new approach to the interpolation of values for incomplete precipitation maps. The authors motivate their study by explaining that such interpolation is often necessary as result of various factors, such as radar outages. They explain the structure of their “channel-based” approach, combined with an LSTM module, and apply it to a dataset provided by Germany’s national meteorological service. Finally, they show via a range of metrics and modeling scenarios that their proposed approach may outperform more basic ones.
The overall quality of the writing, referencing, structure and experiments is good. There do not appear to be any notable and relevant works missing from the list of references. Although not providing ground-breaking contributions to the field, the results are nonetheless valuable in illustrating the feasibility of alternative approaches to the problem of precipitation map in-painting using temporal data. The description and review of the addressed problem is also good. The manuscript would be suitable for publication in the journal after minor revisions to address the following concerns and questions. Particularly notable insightful and clear parts of the manuscript are also noted below.Response: we thank the reviewer for the review and for these encouraging remarks. Regarding novelty, we do believe that this approach is rather new, as the ML method allows us to 1) take account of moving weather systems in a dynamical fashion, 2) were able to update the entire existing catalogue of the German weather service, and 3) provide a potential method to quickly infill rainfall information when radar systems fail, for instance in emergency situations. We will add text to the paper, to stress these points.
## Review ratings
Scientific significance: Good
Scientific quality: Good
Presentation quality: Fair
## Specific and technical comments
Line 24: Unclear if this is in fact referring to Figure 1.Response: indeed, we meant to refer to Fig. 1, we will adjust this in the revised paper.
Line 40: More detail should be provided to explain what is meant by “data-driven”. I assume such methods are machine learning adjacent ones which contrast to statistical methods driven by modeling assumptions, but additional clarity would be helpful here.Response: we will add the difference here between numerical models and/or statistical and geoinformation techniques based on fixed equations, rather than learning algorithms based on large datasets where the equations are learned by training (which is “data-driven”).
Line 47: Additional detail would be beneficial here. It appears unfair to state that data-driven methods only account for spatial relationships while statistical methods only account for temporal variability. Explanation of the caveats of this statement should be included. For example, why is spatial interpolation, a statistical method, unable to reconstruct spatial features?Response: we meant to indicate that temporal changes in parts of the spatial data (images) are used to learn about infilling in missing parts. This is the case with “moving” weather systems that drive rainfall patterns. This is a feature that is not included in geostatistical methods. We will adjust this sentence to make that clear.
Line 59: The explanation of the dataset is clear and well organized. However, it is unclear which exact time period is used for this study.
Response: we will add the years for which we used and updated the data, which is the period 2001-2022.
Line 85: Please provide a citation for this statement.
Response: We added Liu et al. 2018, where they compare their partial convolutions with other techniques and show an improvement of their technique.
Equations 1: Please use \mathrm to format the text
Response: We will correct this.
Line 102: This paper seems to be a seminal work in the field, and the overall approach of the presented work appears rather similar. Since many readers would already have this particular paper in mind when reading through this manuscript, it would be ideal to mention this paper earlier in the introduction and explain its similarities and differences with this work.
Response: Thanks for this suggestion, and we agree. We added a sentence in the introduction, where we explain that our method is a combination of the work from Liu et al. 2018 and Shi et al. 2015, combining partial convolutions and convolutional LSTMs.
Line 122: There should be a brief mention of the programming language and packages used to implement the model, hardware utilized, and run time here.
Response: we thank the reviewer for this comment. We added a “Technical Implementation” subsection in the methodology section, where we go into detail here. We added training and inference times in the discussion section.
Line 125: The experiments and associated results in this section are highly yet appropriately detailed. However, it is difficult at times to follow what is going on. Some reorganization here could be beneficial to the reader’s understanding. For example, the results and discussion could be split into at least two different sections.
Response: We added a subsection “Experimental Design” to the methodology section and moved all parts that describe the experiments to this. We hope this is beneficial to the reader’s understanding.
Line 137: Please use \mathrm to format the text
Response: We will correct this.
Figure 2: This is a good way to present the relevant metrics in a visually appealing and informative manner
Response: we thank the reviewer for this compliment.
Figure 7: The text in this figure is difficult to read as it is too small.
Response: we will check the font size and also ask the journal about requirements.
Line 198: The conclusion is clear and provides a high-level summary of the main findings. A very brief reiteration of the motivation for the study here would also be helpful.
Response: we will add a reiteration for the scope and motivation of the study in the revised paper.
Citation: https://doi.org/10.5194/egusphere-2024-1392-AC1
-
AC1: 'Reply on RC2', Johannes Meuer, 19 Feb 2025
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
392 | 115 | 32 | 539 | 24 | 27 |
- HTML: 392
- PDF: 115
- XML: 32
- Total: 539
- BibTeX: 24
- EndNote: 27
Viewed (geographical distribution)
Country | # | Views | % |
---|---|---|---|
United States of America | 1 | 191 | 35 |
Germany | 2 | 168 | 31 |
China | 3 | 25 | 4 |
India | 4 | 14 | 2 |
France | 5 | 13 | 2 |
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1
- 191