the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Global Sub-national Impact-based Forecasting for Tropical Cyclones Using Open Data: Combining Machine Learning and Exposure-based Approaches
Abstract. Tropical cyclones (TCs) cause substantial and uneven impacts across regions, driven by differences in exposure and vulnerability. While anticipatory action (AA) systems aim to mitigate these impacts, they are typically based on hazard thresholds rather than predicted consequences, limiting their effectiveness and consistency. Impact-based forecasting offers a promising alternative, but existing approaches are often region-specific or rely on non-transferable data. In this study, we develop a global, sub-national impact-based forecasting framework that predicts affected-population fractions using only openly available data. The model integrates hazard, exposure, and contextual features within a two-stage XGBoost architecture and is evaluated across 780 historical TC events using decision-relevant metrics aligned with operational thresholds. Our results show that machine learning improves the detection and spatial localization of impacts, but does not outperform simpler exposure-based approaches in identifying severe events. This reveals a fundamental trade-off between coverage and conservative severity detection, suggesting that hybrid strategies combining both approaches are better suited for operational use. We position this system as a first-generation global benchmark for impact-based forecasting: it demonstrates the feasibility of transferable, sub-national predictions using open data, while clarifying the limitations that must be addressed for reliable deployment in anticipatory action systems.
- Preprint
(3910 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2026-1996', Bernard Alan Racoma, 24 Jun 2026
-
AC1: 'Reply on RC1', Kyriaki Kalimeri, 07 Sep 2026
We would like to thank the reviewer for the careful reading and constructive feedback on the abstract. Please find below our replies to each raised point.
Methods comments:
Comment 1
"The manuscript refers to IBTrACS for cyclone track and intensity, but does not specify which version or variables were used... If the default IBTrACS combined best track was used, please state this explicitly and discuss any limitations arising from inter-agency differences in the underlying data."
We thank the reviewer for identifying this reproducibility gap. We will specify IBTrACS v04r01, the CLIMADA version used, and the input variables: track positions, central and environmental pressure, and maximum sustained wind speed. We will describe the default provider-selection sequence used by TCTracks.from_ibtracs_netcdf() when no provider is specified, including the official-agency data and fallback agencies, and the rescaling to a common 1-minute sustained-wind equivalent. We will also discuss residual inter-agency differences in intensity-estimation methodologies and post-season reanalysis that this conversion cannot remove.
Comment 2
"Please clarify: are the TC tracks EM-DAT or IBTrACS? Additionally, please include time period, or inclusive years."
The tracks shown in Figure 2 are IBTrACS best-track geometries (the source used for the track-derived wind predictor), restricted to the subset of events that are also reported in EM-DAT, and they cover 2000–2022, the same window as the main analysis. We will state this explicitly in the Figure 2 caption and reiterate the study period in the Methods.
Comment 3
"The description of the two-stage XGBoost model in Section 3.2 is clear in terms of structure, but its operational use is somewhat unclear... The paper would benefit from a more explicit step-by-step discussion of the said figure. As currently written, it is not immediately clear whether the model is intended as a retrospective predictor or as an operational forecasting tool. Clarifying this distinction and explicitly describing the prediction timing (e.g., pre-landfall lead times) would improve readability."
We will expand the explanation of the existing two-stage schematic (Figure 3) and link it to the end-to-end pipeline (Figure 1): assemble predictors, classify impact occurrence, estimate the affected fraction for cells classified as impacted, and aggregate predictions to ADM1 for evaluation. We will distinguish the retrospective best-track evaluation from the separate track/wind sensitivity experiment at lead times from 72 hours before landfall to landfall. In that experiment, only the track-derived wind input is forecast-based; observation-derived rainfall is held fixed, including rainfall unavailable at forecast issuance. We will therefore describe it as an exploratory sensitivity experiment, not an operational forecast validation.
Comment 4
"The manuscript fixes the classification threshold at 0.5, but it is unclear whether this choice was empirically justified. Given the class imbalance and decision-oriented context, please describe how this threshold was selected and whether other values were evaluated."
We thank the reviewer for raising this. The manuscript originally fixed this threshold at 0.5 without an empirical justification. The additional development search considered values from 0.30 to 0.70 jointly with the other hyperparameters and selected 0.5 within that search. We will report the search space, selection criterion (ADM1-level F1 at the 15% impact threshold), and selected configuration in the revised manuscript and supplementary information. Because development records were reused in the subsequent leave-one-event-out evaluation, this was not nested tuning and the evaluation may be optimistic. We will describe 0.5 as the value selected under this search and objective, rather than as a generally optimal threshold.
Comment 5
"The manuscript refers to a 'reported impact fraction' and uses thresholds of 0% and 15%, but the definition of this quantity and its operational meaning are not immediately clear... Consider explicitly defining 'impact fraction' at first mention... and restating how the 0% and 15% thresholds map to operational categories."
We will define the reported affected-population fraction explicitly at first mention in Section “Model Architecture" (Stage 1 bullet) as the number of people reported affected divided by the population of the corresponding reporting area, tying it directly to the affected-population fraction defined in Section “Event History and Target Construction”, and clarifying that the same underlying quantity is used to define both the 0% and 15% thresholds. For evaluation, any impact means a fraction strictly greater than 0; high impact means strictly greater than 0.15 at ADM1 level. The remaining positive fractions, 0 < f <= 0.15, form the low-impact category. These are study decision categories, not validated universal activation rules. We will distinguish these impact-fraction cutoffs from the first-stage classifier probability cutoff of 0.5.
Comment 6
"The choice of 15% threshold is reasonably motivated... However, at this point it remains somewhat merely descriptive and qualitative. For example, 'meaningful separation' [is] not quantitatively substantiated. It would strengthen the manuscript to support this choice with empirical analysis, such as reporting class distributions or model performance across alternative thresholds."
We will add an appendix reporting precision, recall, F1, and observed class counts for the two-stage model across impact-fraction thresholds from 1% to 31%. We will use the same common ADM1-event evaluation sample and corresponding out-of-fold predictions as the main results, and reconcile the sample size and class-prevalence calculations before reporting numerical conclusions. We will present 15% as a policy-motivated study threshold, rather than a globally validated or empirically optimal cutoff. The sweep will show sensitivity to this choice; it will not establish robustness of model rankings unless the baselines are also evaluated at the same thresholds.
Comment 7
"For reproducibility, please provide details on how the XGBoost hyperparameters were determined, including any tuning procedure (e.g., grid/random search and validation strategy)."
We will document both tuning steps: selection of the under-sampling ratios on 50 development records, followed by a random search over 80 configurations on a 200-record development subset, with 160 records used for grouped stratified 5-fold cross-validation and 40 reserved for a final check. We will give the candidate distributions, random seeds, objective (ADM1-level F1 at 15%), selected settings, and the grouping identifier. The development data were reused in the later leave-one-event-out evaluation, so this is not a fully nested estimate and may be optimistic. We will make that limitation explicit, including its implications for the temporal analysis if hyperparameters were selected using later events.
Comment 8
"Section 3.4 would benefit from more detailed documentation of the forecast dataset... it is unclear which forecasting systems (beyond KMS) and regions are represented... the forecast evaluation appears to rely on a relatively small and potentially regionally constrained subset of events... I encourage the authors to explicitly acknowledge this limitation... the description of the TIGGE forecasts (e.g., 'up to 25 ensemble members') is somewhat vague."
We thank the reviewer for these points. We will specify that only the Korea Meteorological Administration's (KMA) contributed tracks were used from TIGGE's multi-center archive, and explicitly list the seven countries represented (the Philippines, Japan, Taiwan, Vietnam, China, South Korea, and North Korea ). All 26 matched country-event records fall within the Western North Pacific basin and cover 2012- 2020. We will distinguish these records from unique physical cyclones.We will document the nominal configuration (12-hourly issuance, 6-hourly valid times to 120 hours, up to 25 members) and the actual available member counts and missing forecasts by lead time.The experiment will be described as a single-centre, single-basin track/wind sensitivity analysis with observation-derived rainfall held fixed; its coverage and performance cannot be assumed representative of other forecast systems or basins.
Comment 9
"The manuscript relies heavily on ADM0 and ADM1 units for aggregation and evaluation, but these terms are only briefly defined... Please provide a clearer definition at first use and consider briefly reminding the reader in later sections. Additionally... it would be helpful to justify this decision and discuss its limitations, as ADM1 units vary considerably across countries..."
We will define ADM0 as country level and ADM1 as the first administrative level below a country, with reminders where needed. We will justify ADM1 as the common reporting and evaluation scale after harmonizing heterogeneous impact reports, rather than imply that all targets are independently observed ADM1 totals. We will explain how reports spanning several administrative units are allocated. We will also discuss how differences in ADM1 area, population, and reporting quality affect aggregation, the meaning of the 15% cutoff, and cross-country comparability.
Results comments:
Comment 10
"Figures 4 and 5 later in the manuscript provide useful information on class distributions and confusion matrices. However, it would be helpful to explicitly summarize the number (or proportion) of observations at each decision threshold (0% and 15%) in the main text or tables. This would make it easier to interpret the reported precision/recall metrics, particularly given the strong class imbalance in the data."
We will add the class counts and proportions in the main text: 25,185 no-reported-impact pairs (85.6%), 3,624 low-impact pairs (12.3%), and 618 high-impact pairs (2.1%), totaling 29,427 ADM1-event pairs. We will reconcile these counts across the figures, tables, and threshold-sensitivity analysis.
Comment 11
"It would also be helpful here if the authors mention how many events were detected at the 15% threshold, beyond listing down that 'observations are rare.'"
We will distinguish observed high-impact regions from correctly detected ones. The current evaluation contains 618 high impact ADM1–event pairs (2.1% of 29.427) ), of which the two-stage model correctly detects 309 (recall 0.50). We will also report the number of country-event records containing at least one high-impact ADM1 unit (136 of 780), separately from the number of unique physical cyclones.
Comment 12
"It would be helpful if the authors mention the number of events at the 0% threshold."
We will add that 4,242 of the 29,427 ADM1–event pairs (14.4%) report some level of impact, while the remaining 25,185 (85.6%) have no reported impact.
Comment 13
"The use of precision, recall, and F1 is appropriate given the class imbalance; however, the relatively small number of high-impact cases (>15%) means that these metrics—particularly F1—may be sensitive to small changes in predictions. It would be helpful to explicitly quantify the number of high-impact observations and discuss how this affects the robustness and interpretation of results."
We will explicitly quantify the high-impact class imbalance. We will add event-level context in Section "Detecting high-impact regions (15% threshold)": of the 780 country-event records evaluated, only 136 (17.4%) contain at least one high-impact ADM1 region. We will distinguish country-event records from unique physical cyclones and discuss the resulting dependence and small effective sample size. We will report storm-clustered uncertainty intervals for the main metrics and explain that high-impact F1 and model rankings are sensitive to the small number and composition of severe cases.
Comment 14
“I appreciate the inclusion of the geo-constrained specification, which assesses how well the model generalizes when trained only on basin-specific data. This is a useful test of spatial transferability and complements the temporal analysis. Given the importance of these findings—especially the insights regarding generalization across basins and the limited benefits of geographically constrained training—I would encourage the authors to consider moving the geo-constrained results (currently in the appendix) into the main text. Doing so would improve the visibility of these results and further strengthen the overall narrative of the paper."
Table of performance by basin (Table C1) and its accompanying discussion will be moved from the appendix into Section “Sensitivity Analysis”, immediately following the paragraph introducing the geo-constrained approach. We will describe this as a comparison of pooled multi-basin and within-basin training, with results stratified by test basin. Because the pooled model can train on other storms from the test basin, this does not test transfer to a completely unseen basin. We will avoid interpreting it as such.
Comment 15
“The title of Section 4.4 (‘Evaluation with Historical Forecasts’) may be somewhat misleading, as the analysis is based on a limited sample (26 events) and is framed as a proof-of-concept rather than a full evaluation. The authors may consider revising the section title to better reflect its exploratory nature (e.g., ‘Proof-of-concept evaluation’ or ‘Exploratory analysis under forecast uncertainty’)”
Will change section name to “Proof-of-concept evaluation with Historical Forecasts”
Comment 16
"The results suggest that the XGBoost model performs particularly well in rainfall-driven or multi-hazard events, whereas wind-based thresholds tend to underperform. However, it remains unclear whether this improvement is due to the use of machine learning itself or simply the inclusion of additional predictors such as rainfall. It would therefore be helpful to acknowledge this distinction and discuss the possibility that simpler multi-hazard or rainfall-based threshold approaches could capture some of these improvements."
Section "Case studies" (Section 4.5) will acknowledge that the two-stage model's advantage in rain-dominated cases may come from its use of rainfall and hazard-interaction predictors rather than the machine-learning component per se, and note that a simpler multi-hazard or rainfall-threshold rule could plausibly achieve similar gains.
Conclusions Section:
Comment 17
"The opening sentences of the Conclusions are clear but somewhat compressed... I recommend expanding this description slightly to more explicitly state the prediction target (e.g., affected-population fractions), the spatial scale (grid-level aggregation to sub-national units), and the two-stage structure (impact occurrence vs. severity). In addition, it would strengthen the Conclusions to more clearly highlight the novelty of the proposed framework..."
Section "Conclusions" will explicitly state the prediction target (affected-population fraction), the spatial scale (0.1° grid, aggregated to ADM1), the two-stage structure (occurrence classifier, severity regressor), and the framework's novelty (two-stage architecture combined with fully open global data, evaluated at decision-oriented thresholds across a large multi-country, multi-basin dataset).
Comment 18
"The future work discussion is relevant but somewhat general, and could be strengthened by more explicitly linking it to the specific findings and limitations identified in the manuscript... potential directions could include hybrid approaches that combine model-based detection with threshold-based severity triggers, further exploration of multi-hazard or rainfall-based thresholds, or strategies to address the scarcity of high-impact events."
We will link future work to the findings and limitations: the contrasting detection and severity error profiles motivate evaluating a prespecified hybrid rule against its components on the same held-out events; unequal predictor sets motivate a predictor-matched statistical baseline and rainfall or multi-hazard rules; and the scarcity of severe cases motivates improved data coverage and explicit uncertainty analysis. We will state that the hybrid rule has not been tested.
Citation: https://doi.org/10.5194/egusphere-2026-1996-AC1
-
AC1: 'Reply on RC1', Kyriaki Kalimeri, 07 Sep 2026
-
RC2: 'Comment on egusphere-2026-1996', Anonymous Referee #2, 03 Jul 2026
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2026/egusphere-2026-1996/egusphere-2026-1996-RC2-supplement.pdf
-
AC2: 'Reply on RC2', Kyriaki Kalimeri, 07 Sep 2026
We would like to thank the reviewer for the careful reading of the manuscript and for the constructive comments. We have addressed each point below.
Abstract comments:
Comment 1
"The abstract adequately summarizes the work, but the claim of being a "first-generation global benchmark" is strong and not sufficiently justified against existing global IBF systems (e.g., CLIMADA-based frameworks already operate globally)."
We agree and will remove the “first-generation global benchmark” formulation altogether. The abstract, introduction, and conclusions will describe the contribution as a global, multi-basin retrospective benchmark of a trained affected-population model against transparent, operational exposure-based baselines, a formulation specific to what the study demonstrates, rather than an unqualified novelty claim among global IBF systems.
Comment 2
"Briefly describe the improvement of the approach in line 8, for example, by providing at least one quantitative metric illustrating the improvement."
We will add the detection F1-score comparison (0.42 vs. 0.27 for the best exposure-based baseline) to substantiate this claim.
Comment 3
"Line 6: for XGBoost, spell out the full name first, then give the abbreviation."
Correction — "Extreme Gradient Boosting (XGBoost)" will be spelled out on first use.
Comment 4
"In line 12, briefly describe the limitations."
We will add a brief mention of the two main limitations (heterogeneous impact-reporting data and limited vulnerability representation), which are discussed in full in the Discussion and Conclusion.
Comment 5
"Provide a clear description of the decision-relevant metrics and operational thresholds."
We will briefly identify precision, recall, and F1 in the abstract and define the evaluation cutoffs as any reported impact (>0%) and high reported impact (>15% of ADM1 population). We will retain the full metric definitions in the Methods and validation-metrics appendix, and make clear that these thresholds are study evaluation categories rather than globally validated activation criteria.
Comment 6
"It is not clear why the machine learning approach reveals a fundamental tradeoff between coverage and conservative severity detection. Add one sentence to explain this."
We will describe this as an empirical trade-off among the tested specifications, rather than a fundamental property of machine learning. At the any-impact threshold, the two-stage model attains higher recall and lower precision than the 33 m/s wind-exposure baseline. At the high-impact threshold, neither approach achieves both high precision and high recall. We will avoid attributing this contrast solely to the algorithm, because predictor sets and threshold choices also differ.
Introduction comments:
Comment 7
"It would be better to combine the Introduction and Related Work sections... In addition, some content is duplicated across both sections, such as lines 45–46 and lines 68–69."
We agree and will merge the two sections into a single Introduction, integrating the Related Work content after the motivation and gap-identification paragraphs and before our contribution statement. We will also remove the duplicated sentence in the former Related Work section (echoing our main finding), retaining the fuller statement in the closing paragraph.
Comment 7a
"Incorrect citation on line 16... cited paper is mainly about anticipatory action rather than cyclone damage."
Correction. The citation to Chaves-Gonzalez et al. (2022), "Anticipatory action: Lessons for the future," will be moved to the anticipatory-action sentence where it belongs, and the TC economic/infrastructure damage claim will be supported by Mendelsohn et al. (2012), "The impact of climate change on global tropical cyclone damage," and Hallegatte et al. (2016), Unbreakable: Building the Resilience of the Poor in the Face of Natural Disasters.
Comment 7b
"For line 18, the following should be cited: Adams et al. (2021)..."
Will add Adams et al. (2021), "Mortality from forces of nature among older adults by race/ethnicity and gender," and use it for the broader statement on unequal disaster mortality among older adults. We will retain a cyclone-specific source for any statement specifically about excess mortality after tropical cyclones.
Comment 7c
"Add citations for: 'AA can reduce losses and improve preparedness,' 'limiting both their effectiveness and comparability,' 'IBF provides a pathway...,' and 'challenges that motivate the need for a consistent global baseline.'"
Citations will be added for all four claims: Chaves-Gonzalez et al. (2022) and Wilkinson et al. (2018), "Forecasting hazards, averting disasters," for the AA-benefits claim; Bierens et al. (2020), on legitimacy and accountability in impact-based forecasting, for the ad hoc trigger inconsistency claim; WMO (2015), “Guidelines on Multi-hazard Impact-based Forecast and Warning Services”, for the IBF hazard-to-consequence claim; and Harrison et al. (2022), "Identifying the impact-related data uses and gaps," together with Arachchige et al. (2025) “A critical review of hurricane risk assessment models”, for the global-baseline motivation.
Comment 7d
"'Many existing models' should be supported with examples and citations."
We will add two concrete examples with citations: single-hazard-dimension models (Mendelsohn et al., 2012; the HAZUS methodology of Vickery et al., 2006), and region-specific vulnerability models (Baldwin et al., 2023; Eberenz et al., 2021).
Comment 7e
"Add citations for 'which impose rigid assumptions and limit transferability.'"
We will support the descriptions of expert-weighted indices and parametric vulnerability curves with appropriate methodological references. We will replace the claim that these approaches impose rigid assumptions and limit transferability with a narrower statement: their performance outside their calibration context requires validation.
Methods comments:
Comment 8
"Lines 90–94 should be in the Introduction, not in the Methods section."
We agree. On review, we found this content substantively duplicates a point already made in the (now-merged) Introduction/Related Work discussion of regionally calibrated models (citing the same Zhejiang CNN study). Rather than duplicating it, we will remove these lines from Methods, preserving the point without repetition.
Comment 9
"From lines 96 to 98, spell out the full names of these abbreviations and add relevant citations the first time they appear."
All dataset abbreviations (IBTrACS, GPM/IMERG, SRTM, GADM, GHSL, JRC, GFDRR, COAST-RP) will be spelled out in full with citations at their first appearance in the Methods.
Comment 10
"On line 155, it is unclear which version of the WorldPop data was used in the study."
We will identify the population product by its full dataset title, reference year, release/DOI, file name (ppl_2020_1km_Aggregated.tif), and download date. The product reported in our processing documentation is the WorldPop 2020 unconstrained, non-UN-adjusted global mosaic at 30 arc-second resolution (approximately 1 km; DOI: 10.5258/SOTON/WP00647). We will correct the former 100 m / UN-adjusted description consistently in Methods, Data Availability, the bibliography, and the feature table.
Comment 11
"No hyperparameter search is reported for the XGBoost model itself. The parameters (max_depth=4, learning_rate=0.01, n_estimators=100) are fixed without justification or tuning. This is a significant omission given the paper's claim of optimizing model performance."
We will document both tuning steps: selection of the under-sampling ratios on 50 development records, followed by a random search over 80 configurations on a 200-record development subset, with 160 records used for grouped stratified 5-fold cross-validation and 40 reserved for a final check. We will give the candidate distributions, random seeds, objective (ADM1-level F1 at 15%), selected settings, and the grouping identifier. The development data were reused in the later leave-one-event-out evaluation, so this is not a fully nested estimate and may be optimistic. We will make that limitation explicit, including its implications for the temporal analysis if hyperparameters were selected using later events.
Comment 12
"On line 230, which administrative unit is referred to—Admin 1 (province/state) or Admin 2 (city)? The difference in geographic resolution can influence the absolute number of affected people."
We will clarify that the evaluation uses ADM1 (the first administrative level below a country). We will explain the Philippine policy motivation for the 15% criterion and the limitations of applying a common fraction across ADM1 units of very different populations and sizes. We will not imply that the Philippine criterion establishes a globally valid threshold for this prediction target.
Comment 13
"Line 346: 'We consider the detection of any impact (0%) and high impact (15%), and explore the error structure magnitude-based and by impact category.' This sentence is unclear and grammatically incomplete."
Correction — will rewrite as two clearer clauses separating the two decision thresholds from the error-structure analysis.
(Proposed manuscript wording) "We evaluate the detection of any reported impact (>0%) and high reported impact (>15%). Separately, we examine absolute errors in predicted affected-population fractions, overall and within each impact category."
Results comments:
Comment 14
“Although the authors claim that the XGBoost model outperforms the wind-exposed model in impact detection, its precision is 0.3 while the wind-exposed model achieves 0.57. Regarding recall, the historical baseline achieves 0.95 while XGBoost achieves 0.68. One could argue that combining the historical baseline and the wind-exposed model would allow both the detection of where impacts occur and the identification of severe impacts.”
We agree that the two-stage model does not dominate every metric: the historical baseline has higher recall at the any-impact threshold, while the windspeed-exposed baseline has higher precision. We will qualify the improvement claim as higher F1 for any-impact detection among the tested models. The windspeed-historical baseline already combines exposure with a historical fraction by multiplication, but it is not the proposed two-part decision rule that uses one signal for detection and another for severity. We will explicitly acknowledge that such a hybrid has not been tested, remove claims that a hybrid is implemented or validated, and describe evaluation of a prespecified combination as future work.
Comment 15
“The confusion matrices (Figure 5) show that the two-stage XGBoost produces 1,238 false high-impact predictions versus 309 true high-impact detections. This false alarm rate is operationally problematic and is not adequately discussed.”
We agree that the false-alert burden needs explicit discussion. Figure 5 gives 1,238 no-impact regions and 554 low-impact regions predicted as high impact, for 1,792 false high-impact alerts in total, versus 309 true high-impact detections. This corresponds to precision 309/(309+1,792) = 0.147: about 85.3% of high-impact alerts are false relative to the reported labels, or about 5.8 false alerts per true detection. The false-positive rate has a different denominator: 1,792/(29,427-618) = 6.2%. We will report this distinction and discuss resource costs, verification needs, and the lack of support for using the model as a stand-alone severe-impact trigger.
Comment 16
"The geographic sensitivity analysis (Table C1) shows that the Australian Region has F1=0.00 at the 15% threshold under both the global and geoconstrained specifications. This means the model completely fails for an entire cyclone basin. Furthermore, line 460 claims that the global model 'captures patterns that generalize across basins,' but Table C1 shows highly variable performance (F1 at 15% ranging from 0.00 to 0.38). This variability undermines the generalization claim. This limitation and the spatial differences across the globe deserve discussion and emphasis in Section 5."
We will state that the current evaluation provides no evidence of reliable high-impact detection in the Australian Region. Its sample contains 15 country-event records, only one with a high-impact region, which makes the estimate particularly unstable; this does not by itself explain away the failure. We will emphasize the basin-level variability at 15% (global-model F1 0.00-0.38) in the Discussion. We will also replace claims of generalization across basins with the more limited conclusion that pooled multi-basin training provides useful any-impact performance within the sampled basins. Transfer to an unseen basin has not been tested.
Discussion Section:
Comment 17
"The discussion of hybrid strategies is conceptually sound but vague. No specific protocol for combining machine learning and exposure-based approaches is proposed or tested."We will state explicitly that no hybrid combination protocol has been tested. We will remove language describing a combined system as implemented, validated, or better suited for operational use. The contrasting error profiles motivate a future evaluation of a prespecified combination against its component models on the same held-out events. Operational objectives can inform that design, but evidence of added value still requires a defined rule and evaluation.
Comment 18
“The WorldPop layer uses only the year 2020 for model training and prediction; however, the target data from EM-DAT span 2000–2022. The differences in population distribution over these 23 years are neither considered in the study nor discussed in the limitations.”
We will report a denominator-only sensitivity analysis using population estimates interpolated between the 2000, 2010, and 2020 WorldPop layers, with extrapolation after 2020. We will describe exactly how affected-region populations and the event-level high-impact label are calculated. Population also enters as a predictor, an allocation factor for grid targets, and an aggregation weight; this analysis does not update the full pipeline or establish robustness of model performance to population vintage.
Our sensitivity analysis reports agreement for 772 of 780 country-event records (99.0%), with eight records changing from below to above the 15% threshold under the year-matched denominator. We will report these counts together with the change in the high-impact class from 136 to 144 records, approximately a 5.9% increase, and the larger differences for early events. These are sample-specific results at country-event level; they do not establish a uniformly downward bias in all locations or unchanged ADM1-level performance.
We will add the fixed-2020 population assumption as an explicit limitation in Section "Discussion," cross-referenced to the sensitivity analysis that will be reported in the new appendix. We will identify a fully year-matched pipeline as a priority methodological extension for future work: this would apply WorldPop's annual layers (2000–2020) consistently throughout target construction (including denominators and spatial allocation), the total_pop predictor, and population-weighted aggregation to ADM1 rather than correcting the denominator alone as done here. We note that a year-matched GHS-SMOD urbanization epoch would address the analogous temporal-mismatch concern for that predictor.
Code Availability:
Comment 19
“Furthermore, the GitHub repository is not available in the Code Availability section for verification.”
The repository link was previously embedded as hyperlinked text ("this repository") rather than shown explicitly. It will be made explicit in the Code Availability section: https://github.com/fedemoss/global-tc-impact-model. We will provide a versioned release corresponding to the reported results and document how to reproduce the main comparisons and the analyses supporting the revision, including the 33 m/s baseline, hyperparameter search, storm-cluster bootstrap, threshold sweep, population sensitivity, and feature ablation. These may be supplied as documented scripts or notebooks, with the required inputs, configurations, and commands; any data-access restrictions will be stated.
Appendix Section:
Comment 20
"Appendix E (SHAP analysis) reveals that prev_events_5years behaves counterintuitively in Stage 1 (higher values → lower predicted impact probability). This is not adequately explained. If regions with more historical events are predicted to have a lower impact probability, this suggests the model may be learning a spurious inverse relationship, possibly due to class imbalance or collinearity."
We will describe the inverse SHAP association cautiously: it may reflect feature interactions, geographic confounding, reporting processes, adaptation, or class imbalance, and the present data do not distinguish these explanations. We will remove any claim that the ablation proves an absence of leakage and will align the feature description, main-text SHAP interpretation, and appendix. We will give the ablation greater prominence: removing prev_events_5years changes any-impact F1 from 0.42 to 0.41 and high-impact F1 from 0.23 to 0.28, while high-impact recall decreases from 0.50 to 0.44. We will present omission as a sensitivity option where reporting bias is a concern, without claiming that it has been shown to improve fairness.
Comment 21
"Appendix F lists 26 forecast events but provides no statistics on how representative these events are of the broader 780-event dataset in terms of intensity, basin, or impact level."
We will summarize the 26 forecast country-event records separately from the unique physical cyclones and describe their basin, intensity, and reported-impact composition. We will define the intensity classification, wind convention, and whether intensity is measured over the full storm lifetime or at country-specific impact/landfall. We will provide corresponding summary statistics for the full evaluation set where available. All forecast records are from the Western North Pacific, so this is a limited single-basin subset and should not be presented as globally representative.
Citation: https://doi.org/10.5194/egusphere-2026-1996-AC2
-
AC2: 'Reply on RC2', Kyriaki Kalimeri, 07 Sep 2026
-
RC3: 'Comment on egusphere-2026-1996', Anonymous Referee #3, 27 Jul 2026
The manuscript provides a useful benchmark and is refreshingly open about its limited skill in predicting severe impacts. However, its main claim of globally transferable, grid-scale forecasting is not fully supported because the grid-level targets are derived by redistributing ADM1 totals rather than from actual local observations.
Recommendation: Major revision
Major comments
- The 0.1° targets are not based on observed local impacts. They are created by distributing ADM1 totals according to population. This means grid-scale skill cannot really be validated, and the study may be better framed as an ADM1-level model with disaggregated outputs. Please provide more discussion on this and maybe reframe some of the statements used throughout the manuscript
- The forecast experiment uses observed IMERG rainfall within ±48 hours of landfall, including rainfall that would not have been available at earlier lead times. This is therefore not a true anticipatory forecast. The authors can explore the use forecast/modeled rainfall or present the experiment only as a track/wind sensitivity test.
- Using 2020 population and 2025 urbanization data for events from 2000–2022 creates a temporal mismatch and possible leakage. Historical or nearest-year data would be more appropriate. Or maybe provide a strong justificatio on the use of 2020 as reference year
- Unreported ADM1 impacts should not automatically be treated as zero, given the uneven reporting in EM-DAT. Add more discussion on the limitations of EM-DAT and differences in cross-country reporting and how that might affect your analysis
- The prev_events_5years variable may partly reflect reporting capacity and could disadvantage data-poor regions. Results without this variable should be given more emphasis.
- The 15% high-impact threshold is not clearly justified for global use. The authors should test other percentage thresholds and consider absolute affected-population thresholds.
- Grid cells with the same ADM1 label are not independent. The analysis should account for clustering and unequal weighting across administrative units.
- Overall performance remains modest, with low precision and many severe impacts missed. Claims about operational use and early warning should therefore be toned down.
- Stronger statistical and hazard-based baselines are needed to show whether XGBoost provides meaningful added value, particularly for operational Impact-based Forecasting.
Minor comments
- Title: Make it clearer that the study is a retrospective benchmark or proof of concept rather than a fully operational forecasting system.
- Throughout the manuscript, especially L15–45 and 95–145: Use “risk,” “hazard,” “exposure,” “impact,” and “vulnerability” consistently, as these terms are not interchangeable.
- L19–21: The statement that climate change is shifting TC occurrence toward previously less-exposed regions is too broad. Qualify it by basin, metric, and level of confidence. “Increasing their intensity” should distinguish among observed changes, projected changes, and changes in the proportion of intense TCs.
- L38–40: “Sub-national (grid) resolution” mixes two different spatial concepts. A 0.1° grid is not an administrative level.
- L45–50: The main finding seems to pre-empt the results and discussion. Consider moving this material to Section 4 or the conclusions.
- L54 and elsewhere: Use the acronym “TC,” since it has already been defined at first instance
- L77–78: Explain why the Philippines is singled out here.
- L93–94: Briefly discuss the limitations of using open datasets, including differences in data collection and reporting systems across countries.
- L96: Spell out IBTrACS, NASA GPM/IMERG, SRTM, and the other datasets at first mention. Also specify the dataset versions used.
- L98–102: Bilinear interpolation is not an aggregation method. Clarify whether the continuous rasters were resampled, averaged, or interpolated.
- L105–114 and 280–284: Define the TC selection criteria more clearly, including the intensity threshold, basin treatment, and EM-DAT–IBTrACS matching procedure.
- L110–114 and 300–307: Use “landfall or closest approach” consistently where both are included.
- L140–144: Revise the statement that socioeconomic vulnerability data are not globally available. Several global proxies exist, so the authors should explain why these were not used.
- L143–144: The manuscript states that 19 of the 72 countries were missing from the SHDI dataset. Explain how this gap was handled.
- L154–157: Using 2020 WorldPop and 2025 urbanization data for events from 2000–2022 creates a temporal mismatch and possible future-information leakage. Historical or nearest-year data should be used.
- L180–200 and 280–284: Clarify how a TC affecting several countries is counted i.e. as one event, several country-events, or multiple ADM1–event observations.
- L238–243 and 263–269: Report the numbers and proportions of no-, low-, and high-impact samples.
- Table 1, around L345–350: “15% (some vs. high impact)” is unclear. Consider “≤15% versus >15%” or “non-high versus high.”
- L285–295 and throughout Section 4: Provide uncertainty intervals for all main performance metrics, not just the rolling results.
- L427–434: Consider bootstrap resampling done at the cyclone-event level to preserve dependence among grid cells and ADM1 observations.
- L507–550: The case studies should not be treated as performance evidence unless their selection criteria were defined beforehand.
- L507–550: Include at least one Philippine failure case, given the country’s large contribution to the dataset.
- L470–478 and Figure 7: Use “Kammuri (Tisoy)” at first mention.
- L470–478: Specify the landfall time and location used for Kammuri/Tisoy, which crossed the Philippine archipelago several times.
- L245–255: Note that the Saffir–Simpson scale is not operationally used by PAGASA or most western North Pacific agencies.
- L245–255 and 509–515: The 33 m s⁻¹ threshold represents hurricane-force winds and may miss damaging tropical-storm-force winds and rainfall-dominated of relatively less intense events.
Citation: https://doi.org/10.5194/egusphere-2026-1996-RC3 -
AC3: 'Reply on RC3', Kyriaki Kalimeri, 07 Sep 2026
We thank the reviewer for the thorough and constructive review of the manuscript. We address each comment in detail below.
Comment 1
"The 0.1° targets are not based on observed local impacts. They are created by distributing ADM1 totals according to population. This means grid-scale skill cannot really be validated, and the study may be better framed as an ADM1-level model with disaggregated outputs. Please provide more discussion on this and maybe reframe some of the statements used throughout the manuscript."We agree this caveat merits explicit discussion and will add it as a limitation in Section "Discussion", together with a reframing of what our evaluation can support: “spatial localization” will explicitly refer to identifying affected ADM1 units; grid cells within an ADM1 are described as repeated labels rather than independent local observations; and the finer-grained grid output will be described as supporting hypothesis generation for within-ADM1 prioritization, rather than validated local impact estimates.
The existing grid-versus-ADM1 comparison supports a trade-off, not validated sub-ADM1 accuracy. The grid-based model has higher recall (0.68 versus 0.14 for any impact; 0.50 versus 0.32 for high impact), but lower high-impact F1 (0.23 versus 0.33) and higher MAE and RMSE (3.67 versus 1.53; 9.52 versus 8.41). We will report these results together and qualify the rationale for retaining grid-based features. Within-ADM1 outputs will be described only as unvalidated exploratory patterns that would require local validation before being used for spatial prioritization.
Comment 2
"The forecast experiment uses observed IMERG rainfall within ±48 hours of landfall, including rainfall that would not have been available at earlier lead times. This is therefore not a true anticipatory forecast. The authors can explore the use of forecast/modeled rainfall or present the experiment only as a track/wind sensitivity test."
We agree this experiment should not be presented as a fully operational anticipatory forecast, and will address this concern through changes made in response to related comments elsewhere in this review: the corresponding Results section will be retitled “Proof-of-concept evaluation with Historical Forecasts”, and the experiment will be consistently described as an exploratory sensitivity analysis of track- and wind-forecast uncertainty with observation-derived rainfall held fixed. In addition, in the Methods section we will no longer describe the ±48 h IMERG rainfall accumulation as information available before event onset: the revised text will state explicitly that this window extends past early forecast issuance times, and that only the track-based wind hazard is replaced by forecast tracks in this experiment.
Comment 3
“Using 2020 population and 2025 urbanization data for events from 2000–2022 creates a temporal mismatch and possible leakage. Historical or nearest-year data would be more appropriate. Or maybe provide a strong justification on the use of 2020 as reference year”
We will explicitly acknowledge that fixed-2020 population and a 2025 urbanization layer do not recreate the information available for events in 2000-2022. Our denominator-only population sensitivity changes the high-impact classification of 8 of 780 country-event records (772 agree), increasing the high-impact group from 136 to 144. We will define this event-level calculation and state that it neither retrains the model nor updates population predictors, grid-target allocation, aggregation weights, or urbanization. Thus, it does not resolve retrospective information availability or establish robustness of ADM1-level predictive performance. We will frame the current analysis as a retrospective benchmark using fixed-vintage covariates and identify a fully year-matched pipeline as a required extension for an operational hindcast.
Comment 4
"Unreported ADM1 impacts should not automatically be treated as zero, given the uneven reporting in EM-DAT. Add more discussion on the limitations of EM-DAT and differences in cross-country reporting and how that might affect your analysis."
We agree that absence of a subnational report is not evidence of zero impact. We will retain zero-filling only as an explicit working assumption and refer to these labels as "no reported impact." We will explain that incomplete spatial reporting can turn genuinely affected ADM1 units into nominal negatives, distort class prevalence, and bias training and evaluation differently across countries and periods. The direction and magnitude of resulting performance bias are not identified by the present data. We will therefore qualify localization and false-alert interpretations against recorded impacts, without asserting that unreported regions are more likely to be truly unaffected.
Comment 5
"The prev_events_5years variable may partly reflect reporting capacity and could disadvantage data-poor regions. Results without this variable should be given more emphasis."
We will give the ablation greater prominence in the main Results and Discussion. Removing prev_events_5years changes any-impact F1 from 0.42 to 0.41, while high-impact F1 increases from 0.23 to 0.28 and precision from 0.15 to 0.20; high-impact recall decreases from 0.50 to 0.44. We will discuss this trade-off and the possibility that the feature captures reporting capacity. The variant without this feature will be presented as a sensitivity option, rather than as a proven remedy for reporting inequity.
Comment 6
"The 15% high-impact threshold is not clearly justified for global use. The authors should test other percentage thresholds and consider absolute affected-population thresholds."
We will evaluate sensitivity to fractional thresholds from 1% to 31% on a consistent full ADM1-event sample, reporting the corresponding class counts and model metrics after reconciling them with the main results. We will present 15% as a policy-motivated study choice, not a universal definition or an empirically validated global optimum. We agree that absolute affected-population thresholds answer a complementary question about the scale of assistance needed. Such thresholds are not evaluated in the current study; we will state this limitation explicitly and describe their evaluation alongside fractional thresholds as future work.
Comment 7
"Grid cells with the same ADM1 label are not independent. The analysis should account for clustering and unequal weighting across administrative units."
We agree that population-based disaggregation repeats the same affected fraction within a reporting unit and does not create independent local observations. We will distinguish three issues: dependence in train/test splitting, dependence in uncertainty estimation, and unequal influence during model fitting. We will document the grouping used for validation and storm-clustered intervals. We will also state that unweighted grid-level training gives units with more retained cells greater influence; event grouping does not correct this weighting. An inverse-cell-count weighting sensitivity could equalize total weight per ADM1-event pair, but has not been evaluated, and we will identify it explicitly as a remaining limitation and future analysis.
Comment 8
"Overall performance remains modest, with low precision and many severe impacts missed. Claims about operational use and early warning should therefore be toned down."
We agree and it will be addressed throughout the manuscript: language around operational readiness will be softened, the forecast-evaluation section will be retitled to reflect its exploratory/proof-of-concept nature, and the trade-offs between recall, precision, and false-alarm rate will be discussed explicitly rather than implied.
Comment 9
"Stronger statistical and hazard-based baselines are needed to show whether XGBoost provides meaningful added value, particularly for operational Impact-based Forecasting."
We agree that the present baselines do not isolate the added value of XGBoost relative to a simpler statistical model using the same predictors, or to a rainfall/multi-hazard threshold rule. Our historical and wind-exposure baselines support comparisons with those specified reference rules only. We will narrow the contribution and superiority claims accordingly, acknowledge the unequal predictor sets, and explicitly identify a predictor-matched statistical baseline and multi-hazard rules as missing comparisons. We will describe these as priority follow-up evaluations rather than claiming that the current experiment establishes an algorithm-specific advantage or operational readiness.
Comment 10
"L98–102: Bilinear interpolation is not an aggregation method. Clarify whether the continuous rasters were resampled, averaged, or interpolated."
We will distinguish resampling from aggregation and report the actual operation for each layer. Where used, bilinear interpolation will be described as resampling continuous intensive variables to the grid. Population counts will be summed using the implemented spatial-overlap rule, and settlement classes will be converted to within-cell proportions. We will retain separate descriptions for other layer-specific means, maxima, or frequency summaries, rather than implying that all continuous rasters underwent bilinear interpolation.
Comment 11
"L105–114 and 280–284: Define the TC selection criteria more clearly, including the intensity threshold, basin treatment, and EM-DAT–IBTrACS matching procedure."
We will describe the study filters explicitly: country-event records in 2000-2022, available subnational reporting, at least 100 reported affected people, and a match to an IBTrACS track, yielding 780 retained records from 922 initial records. We did not impose a separate wind-intensity cutoff or exclude basins at the event-selection stage. The 100-person requirement is our study filter; EM-DAT also admits disasters meeting alternative criteria, including at least ten deaths or an emergency declaration/call for international assistance. We will explain that the EM-DAT-to-IBTrACS correspondence was supplied by OCHA collaborators and document the available matching procedure, ambiguous/unmatched cases, and basin-assignment rule, with a shareable crosswalk where permitted.
Comment 12
"L140–144: Revise the statement that socioeconomic vulnerability data are not globally available. Several global proxies exist, so the authors should explain why these were not used." & “L143–144: The manuscript states that 19 of the 72 countries were missing from the SHDI dataset. Explain how this gap was handled.”We will replace the claim that socioeconomic vulnerability data are not globally available with the narrower explanation that we did not obtain a harmonized subnational vulnerability predictor with complete coverage of our study sample. SHDI lacked coverage for 19 of the 72 countries considered; it was therefore excluded from the feature set, with no imputation and no country exclusions on that basis. We will describe limitations in coverage, spatial resolution, and temporal comparability separately, and acknowledge other proxies, including night-time lights, as untested options.
Comment 13
"L180–200 and 280–284: Clarify how a TC affecting several countries is counted i.e. as one event, several country-events, or multiple ADM1–event observations."
We will distinguish the physical cyclone (IBTrACS SID), the country-event record, and the ADM1-event observation. The 780 event-level observations are country-event records, since a multi-country cyclone contributes separate impact records for each country. We will report the unique-cyclone count and document the grouping identifier actually used for cross-validation, tuning, forecast hold-out, and bootstrapping. We will ensure that the principal physical-cyclone evaluation keeps all country records of one storm together. If country-event grouping allowed the same physical cyclone into training and test sets, we will rerun the affected fitting and evaluation steps and update predictions, metrics, intervals, figures, and conclusions. Any retained country-event analysis will be identified separately as a sensitivity analysis.
Comment 14
"L285–295 and throughout Section 4: Provide uncertainty intervals for all main performance metrics, not just the rolling results."We agree that the main performance metrics require uncertainty intervals. We will report 95% intervals for the principal ADM1-level metrics, including precision, recall, F1, MAE, RMSE, and QWK, using 1,000 bootstrap replicates that resample physical cyclones with all their country and ADM1 observations together. The manuscript draft already contains interval estimates in the main and extended metric tables; we will verify that their grouping follows this definition. These intervals describe variation in evaluated performance across sampled storms, conditional on the fitted out-of-fold predictions. They do not quantify all uncertainty in model fitting, reporting, or future forecasts. Deterministic predictions and nearly constant training-set size do not eliminate uncertainty in estimated performance. Wide intervals for rare high-impact cases are informative and will be reported as a limitation.
Comment 15
"L427–434: Consider bootstrap resampling done at the cyclone-event level to preserve dependence among grid cells and ADM1 observations."
We agree and will use the physical cyclone as the bootstrap cluster, keeping its country records and ADM1 observations together. We will verify that the main metric tables and rolling-window Figure 6 use this grouping, document the cluster identifier and number of replicates, and regenerate any intervals that used finer units. Where validation splits are corrected, we will compute these intervals from the regenerated predictions. The rolling intervals will be described as evaluation-sample variability conditional on the fitted predictions, not as an isolated measure of uncertainty caused by training-data scarcity. We will report the resulting interval widths without assuming in advance that every interval must widen.
Comment 16
"L507–550: The case studies should not be treated as performance evidence unless their selection criteria were defined beforehand."
We will clarify that the four cases were selected after inspecting outputs to illustrate contrasting behaviours, not prespecified as an independent performance test. We will retain them as explanatory examples and remove statements that they establish general superiority by hazard regime or explain the causal source of any advantage. Quantitative model comparisons will rest on the full evaluation; claims about comparative predictive performance within wind-dominated or rainfall-dominated regimes would require a separately defined stratified analysis.
Comment 17
"L245–255: Note that the Saffir–Simpson scale is not operationally used by PAGASA or most western North Pacific agencies." & "L245–255 and 509–515: The 33 m s⁻¹ threshold represents hurricane-force winds and may miss damaging tropical-storm-force winds and rainfall-dominated of relatively less intense events."
We will clarify that 33 m/s is a study wind threshold, approximately hurricane-force for 1-minute sustained winds, rather than a PAGASA operational classification. It excludes damaging lower-wind and rainfall-dominated events by construction. Assigning every resident in threshold-exceeding cells as affected is an extreme exposure-to-impact assumption within those cells, but it is not an upper bound on total multi-hazard impacts. We will qualify all baseline comparisons as conditional on the chosen wind threshold and identify sensitivity to lower wind thresholds as an untested limitation and priority extension.
Comment 18
"L507–550: Include at least one Philippine failure case, given the country's large contribution to the dataset."
We will explicitly identify the Philippine failure example in the case-study discussion and figure caption: Typhoon Krosa/Vinta (2013). In this example, the two-stage model spreads low-impact predictions to neighbouring ADM1 regions and has weaker spatial precision than the windspeed-exposed baseline. We will present it as an illustrative failure mode, not a representative estimate of Philippine performance.
Response to minor comments
We will address the technical corrections in the corresponding manuscript passages. Any additional minor comments not already reproduced above will be listed individually, with the intended correction, so that each reviewer can verify that every point has been answered.
Citation: https://doi.org/10.5194/egusphere-2026-1996-AC3
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 390 | 170 | 34 | 594 | 34 | 33 |
- HTML: 390
- PDF: 170
- XML: 34
- Total: 594
- BibTeX: 34
- EndNote: 33
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
This manuscript presents a timely and well-executed study on global, sub-national impact-based forecasting for tropical cyclones using a two-stage XGBoost framework and openly available data. The work is particularly valuable in its integration of hazard, exposure, and contextual predictors at global scale, as well as its comparison against operationally relevant exposure-based baselines. The inclusion of decision-oriented evaluation (e.g., 0% vs. 15% thresholds), sensitivity analyses (temporal and geographic), and exploratory forecast-based experiments strengthens the contribution and demonstrates careful methodological consideration. Overall, the paper addresses an important problem and provides meaningful insights into the trade-offs between machine learning and simpler rule-based approaches.
At the same time, several aspects of the manuscript would benefit from clarification and further refinement. In particular, some methodological elements would be clearer with additional explanation, including the definition and use of key concepts such as the ‘impact fraction’, the selection of decision thresholds (0% and 15%), and the role of hyperparameters in the XGBoost models. Similarly, certain components—such as the description of the TIGGE forecasts, the operational interpretation of the two-stage framework, and the aggregation to ADM1 units—would benefit from more explicit documentation or justification, especially given the global and heterogeneous nature of the dataset.
Additionally, there are opportunities to strengthen the interpretation and presentation of results. For example, reporting class distributions more explicitly would improve understanding of model performance under strong class imbalance, and the conclusions and future work sections could be expanded to more clearly articulate the methodological contribution, novelty, and implications of the findings (e.g., the complementarity of machine learning and threshold-based approaches, and the role of rainfall-driven impacts).
I have also attached an annotated version of the manuscript with specific comments and suggestions provided directly in the text.
Overall, these are relatively minor revisions focused on clarity, transparency, and positioning, and I recommend the manuscript for publication pending minor revisions.