What do machine learning models learn about floods?

Abstract. Machine-learning (ML) models are increasingly used to estimate flood magnitudes and infer their controls through post-hoc feature importance analysis. However, feature importance does not directly measure hydrological process importance nor infer physical causality. The extent to which feature importance is robust to changes in ML model type (e.g., decision-tree versus neural networks), evaluation setting (e.g., held-out catchments versus held-out years), and predictor information (e.g., static versus dynamic attributes) is an important area of research if ML-based inference is to support flood-generation understanding. Therefore, we systematically evaluate predictive skill and feature importance across 518 catchments in Great Britain, using two ML model types: (i) an event-based Random Forest (RF) trained directly on flood magnitude events; and (ii) a sequence-based Long Short-Term Memory (LSTM) network trained on continuous rainfall-runoff observations, which is subsequently evaluated on the same held-out events as the RF. We test performance and feature importance for the flood magnitude events across a spectrum of evaluation settings, predictor changes, training seeds, and sequence lengths. We assess performance at both the pooled and the within-catchment level, in order to distinguish between inter- and intra-catchment skill. The RF achieved higher predictive skill in every directly comparable model fit (e.g., pooled R² = 0.86 versus 0.68 for the LSTM, under temporal evaluation), reflecting that its event-focused structure is suitable for addressing the task at hand. Notably, for both models pooled skill substantially exceeded within-catchment skill, indicating that much of the apparent skill reflects differences between catchments. The two models failed for different reasons in predicting the largest flood magnitudes, as the RF displayed a magnitude-dependent bias, while the LSTM showed a consistent underestimation. Both models relied predominantly on short-timescale precipitation, with temperature and potential evapotranspiration being utilized over longer timescales. Static attributes performed two separable functions, namely suppressing a baseflow-index structured bias in predicted flood magnitude and shifting antecedent precipitation usage toward shorter timescales. Our findings demonstrate that similar predictive skill can be supported by different feature importance structures, and emphasize the need to test feature importance robustness before inferring process conclusions in future work.