the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 16 Oct 2023
Skill of seasonal flow forecasts at catchment-scale: an assessment across South Korea
Abstract. Recent advancements in numerical weather predictions have improved their forecasting performance at longer lead times for several months. As a result, seasonal weather forecasts, providing predictions of weather variables for the next several months, have gained significant attention from researchers due to their potential benefits for water resources management. Many efforts have been made to link seasonal weather forecasts with Seasonal Flow Forecasts (SFFs) using diverse hydrological models. However, generating SFFs with good skill at finer scales such as catchment remain challenging, hindering their application in practice and adoption by water managers. Consequently, water management decisions, not only in South Korea but also in many other countries, continue to rely on worst-case scenarios and the conventional Ensemble Streamflow Prediction (ESP) method.
This study examines the potential of SFFs in South Korea at a catchment-scale. The analysis was conducted across 12 operational reservoir catchments of various size (from 59 to 6648 km2) over a last decade (2011–2020). Seasonal weather forecasts data (precipitation, temperature and evapotranspiration) from the ECMWF (European Centre for Medium-Range Weather Forecasts, system5) is used to drive a Tank model (conceptual hydrological model) to generate the flow ensemble forecasts. The actual skill of the forecasts is quantitatively evaluated using the Continuous Ranked Probability Skill Score (CRPSS), and it is probabilistically compared with ESP, which is the most popular forecasting system. Our results highlight that precipitation is the most important variable in determining the skill of SFFs, while temperature also plays a key role during the dry season in snow-affected catchments. Given the coarse resolution of seasonal weather forecasts, a linear scaling method to adjust the forecasts is applied, and it is found that bias correction is highly effective in enhancing the skill of SFFs. Furthermore, bias corrected SFFs showed higher skill than ESP up to 3 months ahead, and it was particularly evident during abnormally dry years. To facilitate future applications to other regions, freely available Python packages for analysing seasonal weather and flow forecasts have been made accessible.
- Preprint
(2349 KB) - Metadata XML
-
Supplement
(1201 KB) - BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2023-2169', Anonymous Referee #1, 08 Nov 2023
- AC1: 'Reply on RC1', Yongshin Lee, 15 Jan 2024
-
RC2: 'Comment on egusphere-2023-2169', Anonymous Referee #2, 11 Dec 2023
Summary
This contribution by Lee et al. presents a performance assessment of seasonal flow forecasts generated using ECMWF SEAS5 forecasts and the Tank hydrological model, upstream 12 operational reservoirs in South Korea.
After introducing the experimental setup, the data, the hydrological model and the evaluation framework, the authors analyse the skill of seasonal flow forecasts. First the authors assess the sensitivity of the skill to the hydrological model performance, to the model inputs, namely P, PET and T, and bias correct these inputs. Then they assess the skill of seasonal flow forecasts generated based on SEAS5 with respect to the standard ESP method, distinguishing dry and wet seasons as well as dry and wet years. Lastly the authors show an example of flow forecasts in a given catchment.
Overall, this paper is well structured and written, though several typos remain in some parts of the text, which I tried to list below. The figures are relevant, informative and well presented, though the captions could sometimes be more detailed. The methodology and the analysis of results are both comprehensive and methodical. However, I list hereafter three concerns, the major one being the definition of skill in the manuscript. These are followed by a list of minor comments, mostly asking for clarifications and some reformulating.
Based on this, I recommend the paper for publication subject to major revisions as this work will be a valuable insight into the application of seasonal forecasts over South Korea with an extra focus on reservoir management.
General comments
- The term “skill” in the article is used with different meanings, and sometimes with meanings that are not consistent with the definition commonly used in the literature (see e.g. the books by Wilks, or Jolliffe and Stephenson). The skill in essence is the comparison of the performance of a forecast system with the performance of a benchmark (e.g. ESP is used here) as in Eq. 9 of the manuscript, which I refer to as “skill” in this review. This ratio ranges between -infinity and 1 (see the books by Wilks and Jolliffe and Stephenson for instance). The authors here re-employ the “overall skill” from a previous paper as the percentage of years during which the forecast system has skill with respect to the benchmark, introducing a sense of variability which is interesting. However, in the results section this becomes confusing as the authors use in turn: “actual skill” and “theoretical skill” both having values greater than 1 or 100%, “skill ratio” which is smaller than 1, “relative skill” which ranges between 0 and 100%, “overall skill” which is clearly defined, yet the term is somewhat misleading. It seems important to clarify this aspect for the paper to be understandable, to ensure scientifically sound conclusions. A non-exhaustive list of instances where this was unclear is provided in the detailed list hereafter.
- Related to this first point, the skill thus allows the comparison of two forecasting systems. In section 3.1 it would thus seem natural to look at a skill where the numerator is the CRPS computed against pseudo-observations, and the denominator is the CRPS against real observations (or vice-versa). The result should range between -infinity and 1 if the authors use the skill, or between 0 and 100% if they use the “overall skill”. The same reasoning applies to the comparison before and after bias correction, to the experiment of skill from weather forcings, and to the comparison with the ESP (see for instance the methodologies of Crochemore et al. 2020 and Greuell et al. 2019). However, it was not entirely clear if this is what was systematically done, and I suggest clarifying this for each of the results section and in the figure captions.
- The authors did not exploit much the spatial heterogeneity in catchments, though they do mention that no correlation could be found in terms of skill with respect to the ESP (Section 3.3). I still wonder if explanations could be given regarding the Imha, Buan, and Namgang catchments which stand out in Sections 3.1 and 3.2. The Chungju catchment is also later used to illustrate forecasts. It would be useful to understand the variability that can be found between these catchments to explain differences found in the analysis. Here as well, detailed comments and suggestions are provided hereafter.
Wilks, D. S., 2006: Statistical methods in the atmospheric sciences. Academic Press,.
Jolliffe, I. T., and D. B. Stephenson, 2003: Forecast Verification: A Practitioner’s Guide in Atmospheric Science. John Wiley & Sons Ltd., 240 pp.
Crochemore, L., M.-H. Ramos, and I. G. Pechlivanidis, 2020: Can Continental Models Convey Useful Seasonal Hydrologic Information at the Catchment Scale? Water Resources Research, 56, e2019WR025700.
Greuell, W., W. H. P. Franssen, and R. W. A. Hutjes, 2019: Seasonal streamflow forecasts for Europe – Part 2: Sources of skill. Hydrology and Earth System Sciences, 23, 371–391.
Specific comments
Abstract: There are some typos in the abstract. I suggest some modifications below but invite the authors to screen the text for typo correction.
L14: Replace “to link” with “to generate”
L15-16 “at finer scales such as catchment”: A word seems to be missing
L16: “generating SFFs (…) remains challenging”
L19: “at catchment scale”
L20: “over the last decade”
L23 “actual skill”: this term is not clear at this stage. Please explicit.
L24-25: this sentence states that you compare the skill of forecasts with the ESP. It seems odd. It is rather the ESP which is used as benchmark in the skill computation, and the comparison is carried via the calculation of the skill. Please clarify.
L30-31: are these “openly available”?
L57: this is the first occurrence of ESP, please explicit the term.
L58: “by forcing a hydrological model with historical meteorological observations”
L67-68: “Some of these studies focused”
L95: “did not analyse”
L108: “may be considered”
L110 “on assessing the actual skill and comparing it with ESP”: I assume it is the actual skill of SFFs. This sentence may not work with the definition of the skill: either you compute the skill of SFFs with respect to a benchmark, and compute the skill of ESP with respect to that same benchmark, and then compare both skills, or you directly use the skill for the comparison (its intended use) and choose the CRPS of SFFs and ESP as numerator and denominator of the skill respectively.
Table 1: It would be informative to add Tmin and Tmax to this table, especially given that you have catchments with snow which you later discuss.
Figure 1(d,e,f): Here, instead of showing the average over all catchments, it would be interesting to represent the variability between catchments as it will later inform the variability in forecast skill. In addition, in the caption: L133 “Mean monthly”: isn’t it rather the sum in the case of precipitation, PET and flow?, and L135 “variability of each weather variable”: “of each weather and hydrological variable”. Is it the inter-catchment variability or the inter-annual variability?
L149: Please introduce the abbreviation KMA here.
L156-160: Was the streamflow data generation done as a first step of this work? Do you make a distinction between “streamflow” and “flow” (L156)? After reading this paragraph, I was unsure whether you derived flow values (assuming flow and streamflow refer to the same variable) from measurements of river levels and a rating curve, or from a reservoir water balance, knowing measurements of reservoir levels, inputs other than inflows, and outflows, and then a rating curve. Is it because it is the second option being carried out that reservoir evaporation is mentioned? If so, are you improving on K-water’s method?
L166: Please refer to:
Johnson, S. J., and Coauthors, 2019: SEAS5: the new ECMWF seasonal forecast system. Geoscientific Model Development, 12, 1087–1117, https://doi.org/10.5194/gmd-12-1087-2019.L174: Did you compute PET based on the Penman-Monteith method as mentioned L151, or did you retrieve PET forecasts directly from ECMWF? In the second case, do PET forecasts use the same method as the one used for the historical period?
L176: “45 ensemble members (…) were also selected from (…)” since, in my understanding, there is no generation involved.
L177-180: Here, you first mention the construction of the ESP where each member is simulated, and then mention the parameter estimation of the hydrological model. It might be more intuitive to mention the parameter estimation before mentioning simulations.
L182-184: This sentence shows the issue I have with the “skill” terminology. “The Continuous Ranked Probability Skill (CRPS) method”: the CRPS is a score and not a skill, it stands for Continuous Ranked Probability Score. “method” may probably be removed.
Figure 2: Here the issue with the “skill” appears clearly. Skill should come from the comparison of CRPS values corresponding to two different systems. Here instead, a skill is linked to a single CRPS box, which does not make sense given the definition of skill. In addition, the method used to calculate PET could appear to clarify the point mentioned above.
L222: “a water balance module” and “the United States”
L226: “see Table S1”
L234-235 “higher performance”: what is meant by “performance” here? Each objective function will provide good model performances as long as we focus on the flow characteristics that the objective function focuses on.
L244-247 NSE formulation: the NSE usually compares the simulation to the average of observations and not to the average of simulations.
L260 “the entire range of the parameter of interest”: what do you mean by this? What is the parameter of interest? Do you mean “forecast range”? Please clarify.
L268: This goes against the definition of the skill and of the CRPS. The CRPS alone does not provide an estimate of the skill.
L271-272 “the quality of the skill”: This phrase does not make sense to me. “Quality” is what would be conveyed by the CRPS while “skill” is what is conveyed by the CRPSS. The skill is a ratio of quality/performance indices
L275: “The major reasons”
L283-286 “is more skilful than the benchmark”: A forecast system alone can only have skill with respect to a benchmark. Therefore, we can either say “the system gives higher performances than the benchmark” or “the system has skill with respect to …” (the two being equivalent). Similarly, the forecasting system and the benchmark cannot have the same skill. Lastly, a score of 1 does not necessarily guarantee a perfect forecast, if the benchmark is of sufficiently poor quality.
L287: Usually it is not the CRPSS that is averaged due to the reasons mentioned by the authors. Rather the CRPS that is averaged over all years, and the CRPSS that is computed based on the two averaged values.
L290 “more skillful than the benchmark”: please rephrase
L293 “more skillful than ESP”: please rephrase.
L305: Have you identified a reason for this gap in the last three catchments? Is there a distinctive non-stationary behavior in these catchments? Or are the processes particularly hard to model with the Tank model?
L310 “theoretical skill measured by the mean CRPS”: please rephrase.
L318: It would be interesting to know why this catchment stands out.
Section 3.2: The results shown in Figure 5 are valuable and could help interpret the results of the comparison between SFFs and ESP if it was shown for bias adjusted variables. Figure 6 proves that the sensitivity of the skill to weather forcings is distorted due to biases. Why not show the bias adjustment first and then only the sensitivity to weather forcings so that this analysis can more easily feed the rest of the article?
Figure 5: There is something I do not understand in the results in Figure 5. Assuming that the relative skill represented corresponds to the overall skill resented in the Methodology section, and that the benchmark in the CRPSS is the SFF with all uncertainties (forecasts of P, T and PET). Given that precipitations are key features, replacing forecast precipitation with the observed precipitation (in skill-T and skill-PET of Figure S2) should increase the performance with respect to the benchmark (greater CRPS than that of the benchmark), and should therefore give CRPSS values greater than 0 and an overall skill greater than 50%. Here, the inverse is observed. Could you please clarify this?
L335-337: a word may be missing.
L345: “which in reality”
L348-352: It appears clearly that the skill is degraded in some catchments, for some lead times, which could be fine if the average over years were to increase. However, this is not systematic based on Figure S3. Could the authors please comment on this and maybe add a point in the discussion section or in the Methodology section (L199-201)?
L371-373: The range between 45% and 55% is somewhat subjective. I would recommend applying statistical tests instead to ensure that the full distributions of CRPS are statistically different, for instance.
L403 “average years”: Isn’t Figure 8a showing results for all years, and not only average years?
L408: I suggest “all years” instead of “entire years”
Figure 8: Could you please indicate the number of points that is shown behind the lines? Is it the number of catchments? The number of catchments x the number of years x the number of months in the season? Over how many points is the overall skill computed? Is it statistically representative? Would it be possible to show catchments that stand out and relate it to the analysis in Section 3.3?
Section 3.5: It would help the reader to know the CRPS or skill obtained in this catchment. In addition, an underestimation in wet years and an overestimation in dry years are observed. Could the authors comment on this?
L424: On Figure 9, it seems obvious for the 1-month cumulative flows, but not necessarily for the other time periods.
L462: This point is very relevant. It would be interesting for readers that are not familiar with the area whether ENSO is a good predictor in South Korea.
L496: “useful; however”
L507-508 “we investigated the skill of seasonal weather forecasts”: in my understanding, all analyses focus on the skill of seasonal flow forecasts.
L511: “have not been tested”
L511 “more broadly research”: I am no sure this is correct. Please consider rephrasing.
Citation: https://doi.org/10.5194/egusphere-2023-2169-RC2 - AC2: 'Reply on RC2', Yongshin Lee, 15 Jan 2024
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 261 | 95 | 28 | 384 | 52 | 11 | 14 |
- HTML: 261
- PDF: 95
- XML: 28
- Total: 384
- Supplement: 52
- BibTeX: 11
- EndNote: 14
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1