| 29 Sep 2026
Quantifying the responses of AI precipitation forecast errors to reanalysis errors using analysis of covariance
Abstract. AI weather models are commonly trained by climate reanalysis datasets that inevitably influence AI weather forecasts. Given the importance of precipitation, it is essential to quantify how AI forecast errors respond to reanalysis errors. This paper proposes the use of the analysis of covariance (ANCOVA) method to diagnose the responses. Specifically, extreme precipitation events are identified from historical observations; both forecast and reanalysis errors are calculated against observations during the evaluation period; and the ANCOVA method is fitted to separate the responses to reanalysis errors from changes in response coefficients and additional offsets associated with extreme precipitation. A case study is devised for GraphCast forecasts and European Centre for Medium-range Weather Forecasts (ECMWF) Reanalysis v5 (ERA5), using Daymet observations across 1426 catchments in the Catchment Attributes and MEteorology for Large-Sample SPATially distributed analysis (CAMELS-SPAT) dataset. The results show that the responses of GraphCast forecast errors to ERA5 reanalysis errors weaken progressively with increasing lead time. Under non-extreme conditions, the median response coefficient declines from 0.70 at the 1d lead time to 0.27 at 9d, indicating that each 1 mm/d increase in ERA5 precipitation errors leads to a median increase of 0.70 and 0.27 mm/d in GraphCast precipitation errors, respectively. Extreme precipitation amplifies the underestimation by GraphCast forecasts, with the median additional offset across 1426 catchments decreasing from −1.75 to −10.43 mm/d when the lead time increases from 1d to 9d. Overall, the ANCOVA method contributes to understandings of factors governing precipitation forecast errors and highlights the need to consider inherent uncertainties in climate reanalysis when training and evaluating AI weather models.
The central conclusion, that GraphCast precipitation errors "respond to" ERA5 errors and that this response weakens with lead time, is not supported by the analysis as it is designed. The authors fit a regression (ANCOVA) for each catchment and lead time. GraphCast's daily precipitation error is explained by ERA5's error at the same valid time, an indicator for days above the 90th percentile of Daymet precipitation, and the interaction of the two (Eq. 4). Both errors are computed against Daymet over 1426 CAMELS-SPAT catchments for 2020–2025.
They report that the slope β falls from 0.70 at 1 day to 0.27 at 9 days, and that the extreme-day offset γ falls from −1.75 to −10.43 mm/d. They interpret these results as a weakening inheritance of reanalysis errors with lead time and as amplified underestimation of extremes by GraphCast.
However, the regression coefficients do not measure what the paper says they measure. The design gives β and γ nonzero values even for a forecast that knows nothing about ERA5. The method is a standard regression with an interaction term. The paper's contribution therefore depends on how its coefficients are interpreted. Without controls, the reported trends cannot be separated from ordinary loss of forecast skill or from the structure of the regression itself. Below is my detailed Major comments.
1. A shared observation makes β nonzero without inheritance
Both errors subtract the same Daymet value (Eqs. 2–3; L97–99, L102), which builds a correlation between them into the regression. Because least squares is linear in the dependent variable, each coefficient in Eq. (4) splits exactly into a forecast part and an observation part. For the slope, β = Cov(f, er)/Var(er) − Cov(o, er)/Var(er). Call the first term β_f (the forecast part) and the second β_o (the observation part). The second term, β_o, depends only on ERA5 and Daymet. It is the same for any forecast system at any lead time. As forecast skill decays, β_f goes toward zero, and β goes toward β_o, not toward zero. The decline of β with lead time is therefore expected from skill loss alone. A non-zero β at 9 days is not by itself evidence of inheritance.
2. The extreme-day offset γ is set by how extreme days are selected
Extreme days are defined from the observation itself (Eq. 1, L91), and the observation enters the forecast error with a minus sign. On days selected for large observed precipitation, any forecast that does not fully match will show a large negative error. A perfect forecast gives γ = 0. A forecast with no skill gives γ = γ_o, which is strongly negative because extreme days are chosen for large o. The reported decline from −1.75 to −10.43 mm/d (L317–318) is the shape expected as skill decays between these two limits.
The Discussion attributes the offset to convective and orographic processes, too few extremes in the training data, and spatial smoothing (L335–339). These may well matter, but this analysis does not show them. The authors guard against information leakage when they set the threshold (L82–84), but not against this much larger selection effect.
3. . The analysis cannot separate training inheritance from a shared initial state
GraphCast is initialized from ERA5 (L136). The evaluation period, 2020–2025, lies outside the 1979–2017 training period (L136, L138). The ERA5 errors on these particular valid days were therefore never seen in training. The high 1-day correlation (median r = 0.84, L242) and β = 0.70 can come from three sources: the shared ERA5 initial state; GraphCast reproducing ERA5's systematic relation between atmospheric state and precipitation, which is genuine learned behaviour; or the shared-observation term described in Comment 1.
The paper attributes all three to inheritance from training (L44–45, L329–331, L348–350). It also presents ANCOVA as a cheaper alternative to perturbation and retraining experiments (L345–347). The regression cannot do what those experiments do, because it cannot tell these three sources apart.