the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
evalhyd v0.1.1: a polyglot tool for the evaluation of deterministic and probabilistic streamflow predictions
Thibault Hallouin
François Bourgin
Charles Perrin
Maria-Helena Ramos
Vazken Andréassian
Abstract. The evaluation of streamflow predictions forms an essential part of most hydrological modelling studies published in the literature. The evaluation process typically involves the computation of some evaluation metrics, but it can also involve the pre-processing of the predictions and the post-processing of the computed metrics. In order for published hydrological studies to be reproducible, these steps need to be carefully documented by the authors. The availability of a single tool performing all of these tasks would simplify the documentation by the authors, but also the reproducibility by the readers. However, this requires for such a tool to be polyglot (i.e. usable in a variety of programming languages) and openly accessible, so that it can be used by everyone in the hydrological community. To this end, we developed a new tool named evalhyd that offers metrics and functionalities for the evaluation of deterministic and probabilistic streamflow predictions. It is open source and it can be used in Python, in R, in C++, or as a command line tool. This article describes the tool and illustrates its functionalities using Global Flood Awareness System (GloFAS) reforecasts over France as an example data set.
- Preprint
(2677 KB) - Metadata XML
- BibTeX
- EndNote
Thibault Hallouin et al.
Status: open (until 24 Sep 2023)
-
RC1: 'Comment on egusphere-2023-1424', Barbara Casati, 17 Sep 2023
reply
Dear T. Hallouin and co-authors,
I really enjoyed reading this article, and evalhyd seems a very nice verification too: felicitations! I have uploaded a file with some minor suggestions, while here with this following comment I wish to bring the attention of the scientific community to one aspect of verification, for the general online discussion.
I was particularly triggered about the option in evalhyd of performing conditional verification. I will share here some of our own experience (at the Canadian Met Service) with conditional verification, which maybe can inspire further developments in the tool and, more in general, awareness in the interpretation of the results.
Conditioning on the verification sample can have strong impacts on the verification results (e.g. it can flip the sign of a bias), and hence allows in-depth analysis and understanding of the prediction performance, since the conditioning is usually related to physically-driven phenomena. In a sense, conditional verification is the first step towards process-based diagnostics.
In verification exercises which include several variables (e.g. pressure, temperature, clouds, etc) applying a condition to a variable while verifying a different variable is the common practice (as an example, verification of surface temperature in cloudy versus clear-sky conditions inform of the model performance in reproducing the radiation budget). The condition, however, should be applied to both observed and forecast values (e.g. forecast AND observation being cloudy): I will refer to this double condition as bilateral. On the other hand, when a unilateral condition is applied, to only the observed or forecast variable (e.g. cloudy conditions only for the forecast) this can synthetically introduce a bias in the verification results: in the cloud/temperature example above, stratifying for cloudy conditions only in the forecast leads to a synthetic warm bias for the surface temperature, because in the sample there are bound to be both cloudy and clear sky observations, and when the observations have clear sky the surface temperature is expected to be colder. In other words, the bilateral condition will sample all the “hits” for cloudy sky, whereas the unilateral condition will sample the “hits” and “false alarms” for cloudy sky. From our experience, we advise bilateral over unilateral conditioning. (Of course one can also do unilateral conditioning, but need to be aware of the introduced biases in the interpretation of the verification results).
Applying the unilateral condition to the same variable which is verified might also lead to synthetic biases. As an example, if you stratify your sample for the strong predicted stream flows, you are bound to include in the sample several strong observed stream flows (the “hits”), but also some average or weak observed stream-flows (because the prediction might have some “false alarms”). Then you tend to “artificially” diagnose over-prediction for the strong stream flow (and vice-versa for the low stream flow, conditioning only on the prediction you are bound to find under-estimation, because in your sample you’ll have some observed events which are medium or strong).
I was amazed (also a bit puzzled) to see that in your Figure 6 you have opposite results than I expected (underprediction for high predicted stream flow, more overprediction for low predicted stream flows; the under-dispersion for the average predicted stream flow is instead expected). For me it would be interesting to understand why, is it due to the characteristics of streamflow prediction (where the timing is always predicted well, and hence false alarms and misses are very rare)? what is the behaviour in the other stations? What would you obtain with the bilateral condition?
I would be grateful if you could add in the article some discussion about unilateral versus bilateral conditions.
Thank you + Bests Barbara Casati
Thibault Hallouin et al.
Model code and software
evalhyd: a polyglot tool for the evaluation of deterministic and probabilistic streamflow predictions Thibault Hallouin, François Bourgin https://hal.science/hal-04088473
Thibault Hallouin et al.
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 346 | 99 | 11 | 456 | 7 | 6 |
- HTML: 346
- PDF: 99
- XML: 11
- Total: 456
- BibTeX: 7
- EndNote: 6
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1