the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 08 Jan 2024
Selecting a conceptual hydrological model using Bayes' factors computed with Replica Exchange Hamiltonian Monte Carlo and Thermodynamic Integration
Abstract. We develop a method for computing Bayes’ factors of conceptual rainfall-runoff models based on thermodynamic integration, gradient-based replica-exchange Markov Chain Monte Carlo algorithms and modern differentiable programming languages. We apply our approach to the problem of choosing from a set of conceptual bucket-type models with increasing dynamical complexity calibrated against both synthetically generated and real runoff data from Magela Creek, Australia. We show that using the proposed methodology the Bayes factor can be used to select a parsimonious model and can be computed robustly in a few hours on modern computing hardware. We introduce formal posterior predictive checks for the selected model. The prior calibrated posterior predictive p-value, which also tests for prior data conflict, is used for the posterior predictive checks. Prior data conflict is when the prior favours parameter values that are less likely given the data.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(6374 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(6374 KB) - Metadata XML
- BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2865', Anonymous Referee #1, 03 Apr 2024
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2024/egusphere-2023-2865/egusphere-2023-2865-RC1-supplement.pdf
- AC1: 'Reply on RC1', Damian Mingo Ndiwago, 08 Apr 2024
-
RC2: 'Comment on egusphere-2023-2865', Georgios Boumis, 14 May 2024
I have now finished reviewing the work by Mingo et al. The authors have combined Replica-Exchange Hamiltonian Monte Carlo (HMC) with Thermodynamic Integration in order to do Bayesian inference for the parameters of a conceptual hydrologic model, while simultaneously they compute the marginal likelihood of the model; the latter, facilitates model inter-comparison via the Bayes Factor (BF). In general, the manuscript is well written and has novelty in the sense that the proposed algorithm has never been applied before to hydrological modeling. As a result, I am overall positive! However, I think the manuscript would benefit from a more in-depth discussion (possibly toward the end of the article) about the scientific problem that the authors address, the limitations, and what are some possible alternatives.Â
In light of the extensive comments (major and editorial) of Reviewer #1 with which I completely agree, I would like to raise some concerns about the usefulness of BF as a hydrologic model inter-comparison metric. Please see my comments below:
1. For the synthetic experiments, Tables 4 and 5 show that both DIC and WAIC could correctly indicate the data-generating model, i.e., M2 and M3, respectively. For the average reader, this might practically mean that we do not need BF as an additional metric to "tell" us which model to choose. Please provide an explanation to show why employing BF matters. If you cannot demonstrate that the BF can capture the true underlying model while the other, simpler metrics, cannot, then it is hard to justify your analysis.Â
2. Although I am not a Hydrologist myself, I have a hard time understanding the usefulness of BF within the context of hydrologic model comparison. Traditional hydrologists calibrate models using algorithms like Shuffled Complex Evolution (SCE) based on optimization of a deterministic metric, e.g., NSE. I do understand that Bayesian inference of hydrologic model parameters, on the other hand, is appealing because it naturally provides a measure of uncertainty, which is always important. But the BF provides no pragmatic information to the modeler as per which model is performing better. For example, one would still have to compute NSE or KGE for all models M2, M3, and M4 for the real-world data (Table 8) to get an idea of what's happening. On the contrary, I would argue that for conceptual hydrologic models, which are not computationally demanding and time-intensive, likelihood-free methods like Approximate Bayesian Computing (ABC) might be more suitable for model comparison, as the posterior distributions of parameters for different models are obtained on the basis of an actually useful (to the modeler) distance metric, e.g., NSE, KGE, or even a metric tailored only to river discharge peaks!!!
Again, I am positive about your article and I believe it should be considered for publication, but please provide a better discussion about the practical use of BF as a hydrologic model comparison metric...
Citation: https://doi.org/10.5194/egusphere-2023-2865-RC2 -
AC2: 'Reply on RC2', Damian Mingo Ndiwago, 22 May 2024
We would like to thank the second reviewer for their thoughtful comments. We will address their specific comments in this response and move towards a final response in the coming weeks. We are also more than happy to discuss specific points with the reviewer again. The response is attached.
-
AC2: 'Reply on RC2', Damian Mingo Ndiwago, 22 May 2024
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2865', Anonymous Referee #1, 03 Apr 2024
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2024/egusphere-2023-2865/egusphere-2023-2865-RC1-supplement.pdf
- AC1: 'Reply on RC1', Damian Mingo Ndiwago, 08 Apr 2024
-
RC2: 'Comment on egusphere-2023-2865', Georgios Boumis, 14 May 2024
I have now finished reviewing the work by Mingo et al. The authors have combined Replica-Exchange Hamiltonian Monte Carlo (HMC) with Thermodynamic Integration in order to do Bayesian inference for the parameters of a conceptual hydrologic model, while simultaneously they compute the marginal likelihood of the model; the latter, facilitates model inter-comparison via the Bayes Factor (BF). In general, the manuscript is well written and has novelty in the sense that the proposed algorithm has never been applied before to hydrological modeling. As a result, I am overall positive! However, I think the manuscript would benefit from a more in-depth discussion (possibly toward the end of the article) about the scientific problem that the authors address, the limitations, and what are some possible alternatives.Â
In light of the extensive comments (major and editorial) of Reviewer #1 with which I completely agree, I would like to raise some concerns about the usefulness of BF as a hydrologic model inter-comparison metric. Please see my comments below:
1. For the synthetic experiments, Tables 4 and 5 show that both DIC and WAIC could correctly indicate the data-generating model, i.e., M2 and M3, respectively. For the average reader, this might practically mean that we do not need BF as an additional metric to "tell" us which model to choose. Please provide an explanation to show why employing BF matters. If you cannot demonstrate that the BF can capture the true underlying model while the other, simpler metrics, cannot, then it is hard to justify your analysis.Â
2. Although I am not a Hydrologist myself, I have a hard time understanding the usefulness of BF within the context of hydrologic model comparison. Traditional hydrologists calibrate models using algorithms like Shuffled Complex Evolution (SCE) based on optimization of a deterministic metric, e.g., NSE. I do understand that Bayesian inference of hydrologic model parameters, on the other hand, is appealing because it naturally provides a measure of uncertainty, which is always important. But the BF provides no pragmatic information to the modeler as per which model is performing better. For example, one would still have to compute NSE or KGE for all models M2, M3, and M4 for the real-world data (Table 8) to get an idea of what's happening. On the contrary, I would argue that for conceptual hydrologic models, which are not computationally demanding and time-intensive, likelihood-free methods like Approximate Bayesian Computing (ABC) might be more suitable for model comparison, as the posterior distributions of parameters for different models are obtained on the basis of an actually useful (to the modeler) distance metric, e.g., NSE, KGE, or even a metric tailored only to river discharge peaks!!!
Again, I am positive about your article and I believe it should be considered for publication, but please provide a better discussion about the practical use of BF as a hydrologic model comparison metric...
Citation: https://doi.org/10.5194/egusphere-2023-2865-RC2 -
AC2: 'Reply on RC2', Damian Mingo Ndiwago, 22 May 2024
We would like to thank the second reviewer for their thoughtful comments. We will address their specific comments in this response and move towards a final response in the coming weeks. We are also more than happy to discuss specific points with the reviewer again. The response is attached.
-
AC2: 'Reply on RC2', Damian Mingo Ndiwago, 22 May 2024
Peer review completion
Journal article(s) based on this preprint
Data sets
Magela Creek data (precipitation, discharge, potential evapotranspiration, temperature) D. N. Mingo and Jack S. Hale https://doi.org/10.5281/zenodo.10202093
Model code and software
Selecting a conceptual hydrological model using Bayes' factors Damian N. Mingo and Jack S. Hale https://doi.org/10.5281/zenodo.10202093
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 510 | 178 | 34 | 722 | 28 | 25 |
- HTML: 510
- PDF: 178
- XML: 34
- Total: 722
- BibTeX: 28
- EndNote: 25
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Damian N. Mingo
Remko Nijzink
Christophe Ley
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(6374 KB) - Metadata XML