the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 17 Jul 2025
Global sensitivity analysis of large-scale flood loss models
Abstract. Flood loss models are increasingly used in the (re)insurance sector to inform a range of financial decisions. These models simulate the interactions between flood hazard, vulnerability and exposure over large spatial domains, requiring a range of input information and modelling assumptions. Due to this high level of complexity, evaluating the impact of uncertain input data and assumptions on modelling results, and therefore the overall model “acceptability”, remains a very complex process. In this paper, we advocate for the use of global sensitivity analysis (GSA), a generic technique to analyse the propagation of multiple uncertainties through mathematical models, to improve the sensitivity testing of flood loss models and the identification of their key sources of uncertainty. We discuss key challenges in the application of GSA to large-scale flood models, propose pragmatic strategies to overcome these challenges, and showcase the type of insights that can be obtained by GSA through two proof-of-principle applications to a commercial model, JBA Risk Management’s flood loss model, for the transboundary Rhine River basin in Europe, and Queensland in Australia.
- Preprint
(2489 KB) - Metadata XML
- BibTeX
- EndNote
Status: open (extended)
-
CC1: 'Comment on egusphere-2025-3310', Adam Pollack, 25 Aug 2025
reply
-
AC1: 'Reply on CC1', Francesca Pianosi, 03 Oct 2025
reply
It was nice to see this paper show up on my ResearchGate feed. I greatly appreciate the overview of outcome-based and process-based evaluation of models and the observational challenges for the former. As a reader, I'm more inclined to like a paper that takes this framing. With that in mind, I hope the reviewers and editor will pay attention to a few things that jumped out to me:
Thanks for appreciation of our paper and for contributing to the discussion!1) The manuscript insufficiently reviews and cites relevant literature. For an incomplete set of studies that do some form of global sensitivity analysis for either a full risk-modelling workflow or some component (e.g., inundation modeling) please see:
Tate, Eric, Cristina Munoz, and Jared Suchan. "Uncertainty and sensitivity analysis of the HAZUS-MH flood model." Natural Hazards Review 16.3 (2015): 04014030.
Alipour, Atieh, Keighobad Jafarzadegan, and Hamid Moradkhani. "Global sensitivity analysis in hydrodynamic modeling and flood inundation mapping." Environmental Modelling & Software 152 (2022): 105398.
Hosseini-Shakib, Iman, et al. "What drives uncertainty surrounding riverine flood risks?." Journal of Hydrology 634 (2024): 131055.
Response: Thanks for pointing us to these references. We do not aim to review the (vast) literature of GSA applications to flood hazard models but rather point to the value of extending GSA to the full risk modelling workflow, which to our knowledge is more rarely done - so thanks in particular for the last reference, which is a nice example of the latter!2) At least one citation is incorrect:
Wing, O.E.J., Pinter, N., Bates, P.D. et al. (2022) New insights into US flood vulnerability revealed from floodinsurance big data. Nat Commun 11, 1444.https://doi.org/10.1038/s41467-020-15264-2
should be 2020.Response: Thanks!
3) The discussion of challenges does not adequately recognize the computational bottlenecks involved with running a full global sensitivity analysis that converges. Both application examples run the experiment with few samples. Do the authors know if the risk estimates have converged with such few model runs? I'm not familiar with the PAWN method, but for such a large number of assets under consideration, it would help boost confidence in the reader to show convergence of both risk estimates and sensitivity indices. In some locations, such as the U.S., large-scale studies increasingly use property-level representations of exposure for millions of assets. How should authors/analysts conduct global sensitivity analysis in these settings?
Response: Not sure what is meant here by “convergence” of risk estimates, as risk estimates are – unavoidably – spread out across Monte Carlo simulations (see for example the variability range in the vertical axis of Figure 7). Convergence of sensitivity indices is desirable, though we do not think it is strictly necessary if the inputs ranking is robust to uncertainty in sensitivity indices (which we can quantify for example through bootstrapping). This is discussed in Sec. 4.4 (not for PAWN in particular, but generally for any GSA method) and is one of the solutions that we have found to reduce the computing burden of GSA. Still, our second application example, covering a surface area of about 1.7 million km2 (a fifth of the US) with 300,000 assets, took 3 days on a multi-processors machine - so, GSA application to flood loss models with millions of assets is likely to require a very substantial computational effort and be at the boundary of what is (currently) feasible.4) I notice the authors only consider the PAWN sensitivity index method. Given the article's general framing for global sensitivity analysis for large-scale flood-risk assessment, it seems the article would be more robust and helpful if it considered a wider set of methods in its application examples.
Response: In section 4, where we discuss the challenges in applying GSA to large-scale flood risk models, we point readers to other frugal GSA approaches that could be particularly useful in this context. We can expand this discussion for example by discussing which method (Morris, surrogate-based Sobol’, PAWN, etc.) may be most suitable or become inapplicable in different cases (for example, Morris would not be applicable in our second example because some of the input factors are categorical). However, we do not believe that comparing these methods against each other in the application examples section would add much. The examples are included here with the aim of showcasing the type of results that GSA can yield, and how a flood risk modeller (or a user of flood risk models) would use those results to inform their practice. We believe that what method was used to generate those results is a secondary issue and delving into that aspect a distraction from our main point.5) Why can't the authors make all their data and code available? What would be an unreasonable request to JBA Risk Management? Given the author's invocation of Oreskes et al. (1994), it is ironic to discuss the pitfalls of model evaluation for open systems and then not make foundational materials open and available.
Response: The code for GSA is freely available to all on github. JBA Risk Management’s flood data and models are made available to users under commercial licensing and through collaboration agreements for non-commercial research. We will clarify this in the revised paper. Also, we should clarify the use of the term “open” by Oreskes et al 1994 in reference to systems (=a system is open when its inputs cannot be known in a complete and exact way) so to avoid confusion with the use of the same term in reference to models, data or code.6) Given the use of JBA Risk Management catastrophe model and data, how can the authors declare no conflict of interest? Several authors declare an affiliation with JBA Risk Management Limited and JBA Trust. Even without the explicit affiliation, it is reasonable for a reader to wonder about whether the authors would be able to fully report on limitations in the data and code given the use of proprietary models and data.
Response: We do not believe there is a conflict of interest because some authors are affiliated with JBA Risk Management or JBA Trust. The paper does not aim to promote JBA flood model or benchmark its performance (and in fact we do not - and don’t need to - discuss or make claims on its strengths or limitations). The paper’s aim is to promote the use of Global Sensitivity Analysis, although we are also open about its implementation challenges and limitations, which are discussed in Sec 4 and in the Outlook and conclusions.
Citation: https://doi.org/10.5194/egusphere-2025-3310-AC1 -
CC3: 'Reply on AC1', Adam Pollack, 04 Oct 2025
reply
I really appreciate all the author's comments and hope they recognize that comments 1-5 were aimed to help them understand how a researcher in this space (one who is possibly more inclined to take up their recommendations in my analyses than others in the field) could benefit from refinements to the framing.
Comment 3 meant to point out that it is increasingly common for studies to go higher in spatial resolution and broader in spatial extent - to the extent the authors' goal is to get others in the field to take up these methods, I think they could better achieve the goal by recognizing the contexts in which flood-risk estimation often takes place these days (perhaps those less inclined to do GSA are the ones going broader and broader in spatial extent).
Comment 4 - I apologize for the vagueness of "convergence of risk estimates." I am not nearly the expert in this area as the authors, so I totally defer to them, but I believe there are several desirable convergence criteria when sampling uncertainties. Most importantly, as the authors wrote in the context of GSA, converged sensitivity indices. Robustness of sensitivity index ranking could be one goal (and way to define convergence), but there are other purposes to sensitivity analysis such as factor mapping and this may require other metrics to check for convergence. In the context of comment 3, I think assuming that analysts who most "need" to read this article are not willing to make their workflows more complicated, it could help quite a bit in meeting the authors stated goals.
Comment 5 - again, in the context of me being quite willing to take up the authors' recommendation, the fact that PAWN code is available on GitHub is not particularly helpful for demonstrating to me (and especially others less willing to have more complicated workflows) how to operationalize GSA into our analyses. This was a comment to suggest that because the authors stated goal is for increased uptake of GSA in flood-risk assessment practice, workable examples would really help.
Finally, for comment 6 I'd like to clarify that I do not at all speculate any poor behavior and apologize if my comment came off that way. The NHESS competing interests guidance (https://www.natural-hazards-and-earth-system-sciences.net/policies/competing_interests_policy.html) is quite clear that "A conflict of interest occurs when professional conclusions regarding the complete and objective presentation of research are influenced or could be influenced by a secondary interest. Therefore, we require that our authors disclose such possible competing interests. Competing interests often arise with regard to financial matters. However, conflicts of interest can also be non-financial, professional, or personal and can exist in relation to institutions or individuals." They go on to provide several examples. I went to JBA Risk Management's website and found an article highlighting the preprint here (https://www.jbarisk.com/knowledge-hub/insights/why-exposure-matters-for-flood-risk-uncertainty/). I personally think it is great that the private and academic groups are coming together here for this important paper. However, it is hard to argue that there is not the appearance of potential influence. For one, JBA Risk Management sells services on unverifiable risk estimates (borrowing the Oreskes language). What helps in selling a product that you can't prove is better than your competitors'? Perhaps showing that your data and code gets used in peer-reviewed academic research. And since that data and code is not available for community scrutiny and reuse (comment 5), we have the appearance of a conflict of interest. I find the authors response to my comment fine - I just think transparently stating that along with the article is necessary, as well as consistent with the journal's policies.
Citation: https://doi.org/10.5194/egusphere-2025-3310-CC3
-
CC3: 'Reply on AC1', Adam Pollack, 04 Oct 2025
reply
-
AC1: 'Reply on CC1', Francesca Pianosi, 03 Oct 2025
reply
-
CC2: 'Comment on egusphere-2025-3310', Yukiko Hirabayashi, 08 Sep 2025
reply
General comments
Pianosi et al. examined dominant input uncertainties on flood-induced annual loss, at Rhine River basin in Europe and Queensland state in Australia. They used a JBA Risk Management’s flood loss model to conduct a global sensitivity analysis. Overall, this study represents a contribution to large-scale flood model community and has the potential to provide the model developers to improve model performance by showing the key sources of uncertainty.
I believe that the research methods and the obtained results are described clearly. One of my concern lies in the characterization of uncertainty presented in Table 1. First, I do not fully understand the rationale for assigning a ±50% range to the nonlinear value of the return period. Furthermore, the return periods used in hazard maps and loss calculations are generally predetermined based on river improvement standards as well as the economic and geographical conditions of the site. Therefore, I find it difficult to grasp the reasoning behind discussing uncertainty by varying this value itself.
With regard to the flood event set, I can accept the idea of examining the uncertainty range of the event generation model. However, in determining the ranges of parametric and structural uncertainty in the flood inundation model, I believe that some evidence-based justification should be provided. For example, one might reasonably consider the typical error ranges of global models in precipitation data, evapotranspiration processes, or river discharge calculations.
Since variations in the vulnerability curve directly influence the loss calculation, I believe that the method for determining the range of uncertainty should also be explained with due care. As for the damage ratio, I assume it is determined based on structural types, for instance, distinguishing between above-floor and below-floor inundation. Regarding the lower bound of the depth range, is it assumed to correspond to the height of the first floor? In Table1, DR_min can be set to zero, but does this indicate that, even at the same inundation depth, cases where no damage occurred are taken as the basis for assigning zero damage? Providing explanations for these points would enable a more informed assessment of whether the results are valid.
Finally, the most novel aspect of this study appears to be the simultaneous evaluation of multiple sources of uncertainty. However, because the adopted approach counts only the most influential factor in each comparison, the current method of defining uncertainty ranges reveals that the vulnerability curve contributes the largest share, followed by the flood event set. Yet, the description is insufficient regarding how these uncertainties ultimately affect AAL or LE. For example, if the vulnerability curve were increased by 30% (on the conservative side) and AAL recalculated, by how much would the AAL value change? Would it be by 10%, or by a factor of three? We can see part of the answer from Figure 7 regarding to the AAL in the case of Queensland, but explicitly addressing the magnitude of such differences in the target outcomes would, in my view, greatly strengthen the analysis.
Specific comments
Fig.1: The upper part of Fig. 1 is almost identical to the text and therefore cannot be considered an effective schematic figure. The lower illustration comparing “precise” and “accurate” is easy to understand, but I would suggest reconsidering whether this figure is truly necessary.
Fig.3, caption: I did not find the use of colored boxes particularly effective. In the figure, the hazard map and event set are shown as inputs to the “interpolation” box, but in reality, the flood depth is selected from the chosen event set. Therefore, I believe there may be a more appropriate way of illustrating this process than labeling it as interpolation.
L294: Is there any rationale for setting the value to 0.3? How would the results change if a different value, for example 0.5, were used instead?
L338-349: You have evaluated which factors influence the basin as a whole, but is it appropriate to assess locations that experienced the most severe damage—for example, areas with greater inundation depth—in the same manner as those less affected? For flood-prone areas, might we expect different results, such as certain uncertainties having a greater impact? Since Fig. 6 already organizes the analysis by different return periods, I thought it might be useful to expand the explanation in connection with that figure.
L411-427: Could it be that the aggregation method employed here is problematic? In particular, the fact that the residential portion changes so drastically gives the impression that there may be an issue with the approach itself. If you have any ideas or suggestions for how this could be improved, adding a brief comment on that point would, I believe, be valuable.
Technical corrections
L121: Is “Ls,t” a typographical error for “Ls,k”?
L1599: Is “(FDs,t) a typographical error for “(FDs,k)”?
Citation: https://doi.org/10.5194/egusphere-2025-3310-CC2 -
RC1: 'Comment on egusphere-2025-3310', Francesco Dottori, 07 Oct 2025
reply
The manuscript by Pianosi et al. proposes a framework to analyse the sensitivity of large-scale flood models, taking into account the most relevant sources of uncertainty. In particular, the framework aims at reducing the complexity of the anaysis while maintaining the significance of results.
I appreciated the opportunity to review this work. It's good to see a collaboration between academic and private sectors and I believe the topic is of high interest for the scientific community working on large-scale flood models.
Having said so, I have some remarks mainly regarding the application of GSA to the two case studies.Main points
- The authors do not describe the JBA flood loss model but do provide some references to past works. However, my impression is that the outcomes of the application examples might be influenced by some modelling assumptions (see my following points) and therefore I would suggest including at least a short description of the JBA model.
- Uncertainty in hazard maps: previous studies have shown that hazard maps from large-scale flood models might show limited changes in flood extent and depths across return periods. For example, the 1-in-50-year and 1-in-100-year flood maps might be similar (see Bernhofer et al. 2018, and Aerts et al. 2018). If this is the case for JBA model, this might explain why this parameter has a limited importance in GSA outcomes. Moreover, the authors should explain how flood protection standards are considered in the model, given that they are a major component of hazard map uncertainty (Paprotny et al 2025).
- - Uncertainty ranges in Table 1: the ranges of exposed values and damage functions applied in GSA come from a previous work by Sarailidis (2023). This is fair enough, however, the assumptions and limitations of the study by Sarailidis should be better explained. For instance, my understanding is that Sarailidis obtained the range of damage ratio reported in table 1 by perturbing the values from multiple damage functions found in literature, which in my opionion might lead to overestimate the uncertainty associated to this parameter. Given the outcomes of GSA for the Rhine case study, it is important to communicate that these outcomes might be influenced by this and other assumptions.
Minor points:
- Abstract: The authors begin with "Flood loss models are increasingly used in the (re)insurance sector to inform a range of financial decisions." I would add that flood loss models are increasingly important also in research, (for instant, to understand present and future risk trends as discussed in Ward et al., 2015), as well as in policy support (see for instance the PESETA programme, https://joint-research-centre.ec.europa.eu/projects-and-activities/peseta-climate-change-projects/jrc-peseta-iv/river-floods_en). This would highlight the relevance of the present study beyond the insurance sector.
- One of the main outcomes is the relevance of vulnerability functions in driving model uncertainty, I believe this should be mentioned in the abstract.
- Reference list: missing title for Galloway et al (2025)
ReferencesAerts, J. P. M., Uhlemann-Elmer, S., Eilander, D., and Ward, P. J.: Comparison of estimates of global flood models for flood hazard and exposed gross domestic product: a China case study, Nat. Hazards Earth Syst. Sci., 20, 3245–3260, https://doi.org/10.5194/nhess-20-3245-2020, 2020.
BernhoferBernhofen, M. V., Whyman, C., Trigg, M. A., Sleigh, P. A., Smith, A. M., Sampson, C. C., Yamazaki, D., Ward, P. J., Rudari, R., Pappenberger, F., Dottori, F., Salamon, P., and Winsemius, H. C.: A first collective validation of global fluvial flood models for major floods in Nigeria and Mozambique, Environ. Res. Lett., 13, 104007, https://doi.org/10.1088/1748-9326/aae014, 2018.
Paprotny, D., ’t Hart, C.M.P. & Morales-Nápoles, O. Evolution of flood protection levels and flood vulnerability in Europe since 1950 estimated with vine-copula models. Nat Hazards 121, 6155–6184 (2025). https://doi.org/10.1007/s11069-024-07039-5
Ward, P. J., Jongman, B., Salamon, P., Simpson, A., Bates, P., De Groeve, T., Muis, S., de Perez, E. C., Rudari, R., Trigg, M. A., and Winsemius, H. C.: Usefulness and limitations of global flood risk models, Nat. Clim. Change, 5, 712–715, https://doi.org/10.1038/nclimate2742, 2015.
Citation: https://doi.org/10.5194/egusphere-2025-3310-RC1
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 1,304 | 81 | 20 | 1,405 | 29 | 26 |
- HTML: 1,304
- PDF: 81
- XML: 20
- Total: 1,405
- BibTeX: 29
- EndNote: 26
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
It was nice to see this paper show up on my ResearchGate feed. I greatly appreciate the overview of outcome-based and process-based evaluation of models and the observational challenges for the former. As a reader, I'm more inclined to like a paper that takes this framing. With that in mind, I hope the reviewers and editor will pay attention to a few things that jumped out to me:
1) The manuscript insufficiently reviews and cites relevant literature. For an incomplete set of studies that do some form of global sensitivity analysis for either a full risk-modelling workflow or some component (e.g., inundation modeling) please see: