the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Robustness of the long short-term memory network in rainfall-runoff prediction improved by the water balance constraint
Abstract. While the water balance constraint is fundamental to catchment hydrological models, there is yet no consensus on its role in the long short-term memory (LSTM) network. This paper is concentrated on the part that this constraint plays in the robustness of the LSTM network for rainfall-runoff prediction. Specifically, numerical experiments are devised to examine the robustness of the LSTM and its architecturally mass-conserving variant (MC-LSTM); and the Explainable Artificial Intelligence (XAI) is employed to interrogate how this constraint affects the robustness of the LSTM in learning rainfall-runoff relationships. Based on the Catchment Attributes and Meteorology for Large-sample Studies (CAMELS) dataset, the LSTM, MC-LSTM and EXP-HYDRO models are trained under various amounts of training data and different seeds of parameter initialization over 531 catchments, leading to 95,580 (3×6×10×531) tests. Through large-sample tests, the results show that incorporating the water balance constraint into the LSTM improves the robustness, while the improvement tends to decrease as the amount of training data increases. Under 9 years’ training data, this constraint significantly enhances the robustness against data sparsity in 37 % (196 in 531) of the catchments and improves the robustness against parameter initialization in 73 % (386 in 531) of the catchments. In addition, it improves the robustness in learning rainfall-runoff relationships by increasing the median contribution of precipitation from 45.8 % to 47.3 %. These results point to the compensation effects between training data and process knowledge on the LSTM’s performance. Overall, the in-depth investigations facilitate insights into the use of the LSTM for rainfall-runoff prediction.
- Preprint
(1990 KB) - Metadata XML
-
Supplement
(1341 KB) - BibTeX
- EndNote
Status: open (until 25 Jul 2024)
-
RC1: 'Comment on egusphere-2024-1449', Anonymous Referee #1, 10 Jun 2024
reply
Having evaluated the earlier version/submission of the manuscript, I would like to state that the authors have largely improved the manuscript and clarified most of my concerns and comments. In particular I like the addition of the IG analysis given nice additional insight into the functioning of LSTMs and the role of constrain in this context. So, overall I belief this manuscript provides an interesting and important peace of research from which many readers might profit in their own work.
I have a few questions and comments that should be addressed in a revised version of the manuscript before publication.:
- L15: robustness should briefly be defined, also in the abstract
- L18: The sentence starting “In addition, …” the meaning is not understanding in this formulation, please reformulate.
- L68: Reference for XAI?
- L77: should be “…is passed…”
- L167: Why using catchments with high KGE – please add a reason for that
- L312: should be :”robust”
Citation: https://doi.org/10.5194/egusphere-2024-1449-RC1 -
RC2: 'Comment on egusphere-2024-1449', Anonymous Referee #2, 24 Jun 2024
reply
This manuscript presents a study that attempts to assess the effects of mass conservation constraints on the robustness of LSTM neural networks at a local scale. The study adopts the definition of robustness from Manure et al. (2023), according to which “robustness is the ability to perform consistently across varying conditions.” Based on this definition, the study examines:
- Robustness against data sparsity
- Robustness against parameter initialization
- Robustness in learning rainfall-runoff relationships
I find that this study lacks novelty and that the questions it attempts to answer do not represent a substantial contribution to scientific progress. The manuscript's presentation and clarity could be significantly improved, and some of the results and conclusions are not well justified. The following are some points that I think the authors should consider to improve their contribution:
- The English usage should be reviewed.
- The structure of the paper gives the impression that the authors are merely testing three modeling frameworks and comparing their results. Large portions of the text simply describe their figures, and often the conclusions are case-specific, thus only valid for a selected group of catchments.
- In the section “Robustness against data sparsity,” the conclusions suggest that under certain circumstances, mass conservation constraints can enhance the accuracy of LSTM models. Beyond indicating that more data generally benefits unconstrained models, the study does not provide further information. It would be interesting to identify in which watersheds MC constraints enhance predictions. Additionally, there is a methodological inconsistency as the authors focus their analysis on the mean of KGE values rather than the range or standard deviation of KGE values. Thus, instead of studying robustness, they are analyzing accuracy against data sparsity.
- In the section “Robustness against parameter initialization,” it is indicated that the robustness of LSTM (measured as the standard deviation of KGE values) improves (i.e., standard deviation values decrease) when more data is available for model training. However, this result alone is not very useful. A model's prediction can be consistent but inaccurate. In other words, the standard deviation of the KGE values for a catchment could be small (indicating high robustness), yet those KGE values could still be very poor. Thus, robustness alone is not a very informative metric.
- The section “Robustness in learning rainfall-runoff relationships” could benefit from clearer figures. In this section, the statement “These results indicate that water balance constraint enhances the robustness in learning rainfall-runoff relationships” is not sufficiently supported. The extent to which a model output is explained by one or multiple variables does not indicate whether the model's ability to perform consistently across varying conditions is enhanced. These are two different questions: one is “To what extent do input variables contribute to a model prediction?” and the other is “Does the model prediction vary significantly if it is a function of more variables?”
Citation: https://doi.org/10.5194/egusphere-2024-1449-RC2
Viewed
HTML | XML | Total | Supplement | BibTeX | EndNote | |
---|---|---|---|---|---|---|
142 | 55 | 14 | 211 | 24 | 10 | 8 |
- HTML: 142
- PDF: 55
- XML: 14
- Total: 211
- Supplement: 24
- BibTeX: 10
- EndNote: 8
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1