the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 19 Jun 2026
A PMP-inspired Evaluation Framework for Assessing Deep-Learning Earth System Models
Abstract. In recent years, Deep-Learning Earth System Models (DL-ESMs) have emerged as promising, computationally efficient complements to traditional Earth system models. Here, we present an evaluation framework for testing DL-ESMs from an Earth system model-development perspective using standardized diagnostics from the PCMDI Metrics Package (PMP). This framework allows DL-ESMs, including Ai2’s ACE2 and Google’s NeuralGCM, to be assessed with metrics that quantify their ability to reproduce climatology, major modes of variability, monsoon behavior, and precipitation variability relative to observational reference datasets and CMIP-class benchmarks. By evaluating DL-ESMs with tools commonly used for traditional models, we extend their assessment beyond short-range forecast skill and toward longer Earth system–relevant applications. The results identify encouraging strengths in several large-scale fields and modes of variability while also highlighting persistent challenges in precipitation, tropical variability, and long-run stability for some model versions. This evaluation is a critical step toward building trust in DL-ESMs, guiding future model development, and clarifying their fitness for Earth system science applications.
Status: final response (author comments only)
-
RC1: 'Comment on egusphere-2026-3295', Anonymous Referee #1, 27 Jul 2026
-
AC1: 'Reply on RC1', Giuliana Pallotta, 10 Oct 2026
We thank the anonymous reviewer for the careful and constructive evaluation of our manuscript. The comments identify several places where the initial submission did not sufficiently distinguish diagnostic results from causal interpretation, document the post-processing and stability workflow. We addressed these points through expanded discussions, a new workflow schematic and analysis-design table and revised figures.
Reviewer 1
This study adopts the PCMDI Metrics Package to evaluate the forecasting performance of four deep learning-based prediction models for weather systems at various spatial scales, with intercomparisons against conventional CMIP models. The authors confirm that the AI-based models possess promising predictive skill yet exhibit distinct systematic error features, which they attribute to the limited time span of the training dataset. Although the findings can offer references for the evaluation research of artificial intelligence forecasting models, the analysis lacks sufficient depth. Most discussions merely describe intuitive features shown in the figures without linking them to the inherent disparities among different AI architectures. At minimum, the authors ought to discuss potential causes underlying performance gaps across multiple versions of the NeuralGCM model. Such insufficient and cursory analysis weakens the overall value of this work, and further in-depth elaboration on this part is strongly recommended.
Response: We agree that the initial manuscript emphasized diagnostic description more than interpretation. We have expanded the comparison among ACE2, the original NeuralGCM, NeuralGCM-evap, and NeuralGCM-precip by relating the observed differences to documented architectural and training choices: the learned autoregressive atmospheric state transition in ACE2, the dynamical core and learned physical parameterizations in NeuralGCM, and the hydrologic-closure choices and stochastic formulations of the two precipitation-focused variants. These comparisons suggest possible mechanisms but do not isolate their causal contributions.
Manuscript revision: We added a dedicated Discussion subsection, “Interpreting model differences in light of architecture, training, and sampling” (Sect. 5.1, lines 514–541). For the precipitation climatology, the extratropical modes of variability, the MJO, the monsoons, and precipitation variability across timescales, it relates the empirical result to the relevant model-design difference and presents plausible mechanisms as hypotheses. The structural differences among the configurations are described in Sect. 2.1 (lines 137–148) and Table 1, and the periods and members entering each figure are listed in Table C1.
Specific comments Comment 1.1
“These efforts offer important precedents, but they do not fully address the need for a comprehensive evaluation of DL-ESMs using the same diagnostics routinely applied to CMIP-class Earth System models.”
The authors are advised to clearly respond to their own research question: which extra evaluation metrics are supplemented to enable thorough evaluation of the DL-ESM model.
Response: Thank you for the comments. Our intended contribution is not simply to add more forecast scores, but to test DL-ESMs with climate-oriented diagnostics for long-run climatology, spatial structure, variability, teleconnections, and process behavior. Our evaluation includes annual and seasonal area-weighted climatologies; spatial RMSE and pattern correlation; absolute spatiotemporal RMSE; pressure-level temperature, wind, and geopotential-height fields; teleconnection-mode amplitude and pattern metrics for SAM, NAM, NAO, and PNA; MJO wavenumber-frequency spectra and east–west power ratio (EWR) and eastward power normalized by observations (EOR); monsoon onset, decay, fractional accumulation, hits, misses, and false alarms; temporal precipitation spectra; spatial-variability diagnostics; and Taylor-diagram summaries. Together we believe that these diagnose properties not captured by short-range forecast RMSE or anomaly correlation alone.
Manuscript revision: We use “multi-diagnostic” rather than “comprehensive” in the introduction (Sect. 1, lines 82–84) and explain what the climate-oriented diagnostics test beyond short-range forecast scores (lines 95–107). Table B1 (Appendix B) maps each diagnostic to the model property it tests, its variables, and its reference dataset.
Comment 1.2
PCMDI: Do not assume that all readers are familiar with this abbreviation; its full name should be provided.
Response: We have added the name “Earth System Model Evaluation Project” alongside PCMDI/PMP in the introduction (line 85) to help readers identify the evaluation project and package.
Manuscript revision: PCMDI is spelled out at first use in the introduction (line 85).
Comment 1.3
“We generated ensemble members by varying … For the stochastic NeuralGCM-evap and NeuralGCM-precip configurations, we additionally introduced perturbations through different random seeds.”
What are the underlying reasons for these different ensemble generation approaches? Please provide a detailed explanation of their respective purposes.
Response: The two approaches sample different sources of ensemble uncertainty. For deterministic ACE2 and the deterministic NeuralGCM configuration, members are generated from distinct valid initial atmospheric states and start dates; this tests sensitivity to initial conditions and samples different portions of the prescribed-forcing trajectory. NeuralGCM-evap and NeuralGCM-precip are stochastic configurations, so changing the random seed additionally samples the model-internal stochastic parameterization. The seeds do not perturb the trained model weights. Rather, they generate different realizations of the stochastic fields used by the learned parameterizations, thereby sampling stochastic model uncertainty in addition to initial-condition uncertainty.
Manuscript revision: We added this explanation to Sect. 2.2 (lines 155–164): ensemble construction follows each configuration's documented inference protocol; distinct initial dates sample initial-condition and calendar-state dependence; for the stochastic NeuralGCM-evap and NeuralGCM-precip configurations, independent random seeds additionally sample the stochastic parameterization; and trained parameters are held fixed in every member, with no parameter-space perturbation. Table 1 now gives the ensemble design of every configuration: ten ACE2 members with start dates offset by one or more days; ten NeuralGCM members initialized on the 15th of each month from February to November 1979; ten NeuralGCM-evap members from a December 1979 start date with ten random seeds; and twenty NeuralGCM-precip members from a December 1979 start date with twenty random seeds.
Comment 1.4
Table 2: What is the rationale for selecting the variables in Table 2? This selection serves as one of the key bases for justifying the claimed comprehensive evaluation.
Response: The Table 2 variables were selected to provide a compact but physically diverse evaluation spanning the hydrological cycle, near-surface mass and circulation, and lower- and upper-tropospheric dynamics. Precipitation tests the most parameterization-sensitive hydrologic output; sea-level pressure supports circulation and teleconnection diagnostics; 850 hPa fields sample lower-tropospheric thermodynamic and circulation structure; and 201 hPa fields sample upper-tropospheric jets, waves, and teleconnections. The choice was also constrained by variables available across the DL-ESM archives and by the inputs required by established PMP diagnostics. We agree that this should not be presented as exhaustive Earth-system coverage.
Manuscript revision: We added this rationale to Sect. 2.4 (lines 183–191): the variables were selected for their physical representativeness across levels and processes and their availability across the model archives, with precipitation, sea-level pressure, and the 850 and 201 hPa fields testing the hydrological cycle, circulation and teleconnections, and lower- and upper-tropospheric structure, respectively. The text states that the selection is not exhaustive and that the atmosphere-only configurations preclude ocean and land processes. Table 2 now also lists zg-500, the level used in Figs. 2–3, and marks which variables each configuration provides.
Comment 1.5
“It also introduces a modest generalization test, because the boundary-condition source differs from the ERA5 training environment.”
While this generalization test is more suitable for physical models that strive for physical realism, AI models essentially deliver statistical predictions constrained by existing datasets. The bias correction schemes natively integrated inside the AI model may be incompatible with the boundary conditions set in this experiment. Hence, it remains debatable whether the evaluation metrics can truly characterize the real predictive and simulating performance of the AI model.
Response: We agree that the original phrase “generalization test” was too strong. A change in boundary-condition source can expose sensitivity to forcing-distribution shift, preprocessing, and any learned calibration to the training environment, but it does not by itself establish intrinsic physical realism or out-of-sample forecast skill. We now characterize this experiment as a sensitivity test to boundary-condition distribution shift. The global-mean log-surface-pressure adjustment used for NeuralGCM is a mass-conservation correction rather than a boundary-condition-specific statistical bias correction, but it can still affect derived pressure diagnostics; quantifying this effect requires a separate analysis, which could be included in a follow-on paper.
Manuscript revision: The quoted sentence has been replaced in the new Sect. 3.1, “Boundary conditions and comparison design” (lines 214–217): “Because the prescribed boundary conditions differ from those used in ERA5 training, this setup provides a sensitivity test to boundary-condition distribution shift. While it is not a stand-alone measure of intrinsic physical realism or operational predictive skill, it provides useful information about model performance, reflecting the combined effects of the learned model, forcing compatibility, pre-processing, and any post-processing applied here.” The same framing is used in Sect. 5.2 (line 549), and Sect. 2.3 explains that the log-surface-pressure adjustment is a mass-conservation correction (line 171).
Comment 1.6
Figure 1: Both panels (c) and (d) are titled GPCP3.2. What are the specific differences between them? Do they correspond to different averaging periods?
Response: Panels (c) and (d) originally showed the same observational product (GPCP 3.2) averaged over the different, model-specific comparison periods of the adjacent DL-ESM panels, and the identical titles made these distinct averaging windows unclear. We regenerated Fig. 1 so that both models and the reference are averaged over the same period, January 1980 to December 2014, using GPCP 2.3, whose monthly record covers the whole period. Panels (c) and (d) therefore now show the same reference climatology.
Manuscript revision: Fig. 1 has been regenerated over the common period January 1980 to December 2014 for both DL-ESMs, with GPCP 2.3 as the reference and one realization per model (Table C1), and the caption and the discussion in Sect. 4.1 (line 268) state the common period and the reference product.
Comment 1.7
Figure 4: Why are the results from the other two NeuralGCM model versions not presented?
Response: Figure 4 evaluates pressure-based modes of variability and therefore requires sea-level pressure. NeuralGCM-evap and NeuralGCM-precip do not provide surface pressure or sea-level pressure in the available archives, so the residual-CNN reconstruction used for ACE2 and the original NeuralGCM cannot be applied to those two configurations. Their omission is thus due to a required-variable limitation, not performance-based selection.
Manuscript revision: This limitation is now stated in the footnote of Table 2, in Sect. 2.4 (lines 192–199), at the start of Sect. 4.2 (lines 338–340), and in the discussion of Fig. 4 (line 344). Table 2 marks which variables each configuration provides, and Table C1 lists the configurations and members entering each figure; together they serve as the model-by-diagnostic availability matrix.
Comment 1.8
Figure 6: A more thorough analysis is recommended to interpret the advantages of NeuralGCM, particularly whether the incorporated physical module governs the winter NAO pattern.
Response: We agree that the original discussion was not sufficiently careful. NeuralGCM combines a differentiable dynamical core with learned physical parameterizations, which is a plausible contributor to its representation of large-scale circulation. However, the present comparison does not isolate the dynamical core from the learned parameterizations, training data, forcing, initialization, or sea-level-pressure reconstruction. We therefore cannot conclude that the physical module governs the winter NAO pattern. The revised text discusses the result as an architecture-consistent hypothesis and identifies an ablation experiment as the required causal test.
Manuscript revision: The NAO paragraph of Sect. 4.2 (lines 361–372) now defines NAO performance through the spatial pattern correlation and pattern RMSE, the fraction of variance explained by the CBF mode (44.2 % for ACE2 and 50.6 % for NeuralGCM), and the standard deviation of the associated PC, and it replaces the causal statement with: “The result is consistent with the possibility that NeuralGCM's explicit dynamical core supports large-scale circulation structure, but this interpretation cannot be separated from other model and sampling differences in the present study, including training data, initialization, and the psl reconstruction; isolating the role of the dynamical core would require an ablation experiment.” The Fig. 6 caption identifies the period shown (the members are listed in Table C1) and states that the percentage in each panel title is the explained variance.
Citation: https://doi.org/10.5194/egusphere-2026-3295-AC1
-
AC1: 'Reply on RC1', Giuliana Pallotta, 10 Oct 2026
-
RC2: 'Comment on egusphere-2026-3295', Anonymous Referee #2, 30 Jul 2026
Overview
This manuscript applies the PCMDI Metrics Package (PMP) to four deep-learning atmospheric emulator configurations, and benchmarks them against CMIP6 models and observational references. The premise is correct that forecast-oriented scores are not sufficient to establish whether a data-driven model is fit for Earth-system purposes, and the natural way to close that gap is the diagnostic battery routinely used for CMIP-class models. The breadth of diagnostics is a genuine strength, and several findings are worth publishing.
My concerns are as follows: Several methodological choices are not documented in enough detail to be reproduced or assessed, most importantly the CNN-based ad-hoc machine learning method, which underpins all of Section 4.2, and the stability screening. The comparison against CMIP6 is worth refinement. And no uncertainty is reported anywhere, although the ensembles range from ten members to one.
Note on referencing: the preprint carries no line numbers, so comments below are keyed to page, figure, or section number.
Major Comments
- Scope and novelty. The paper applies an already-published package (Lee et al., 2024, GMD) to four existing models, so the technical novelty needs to be stated and delivered. On my reading, the new elements are (i) the workflow that makes DL-ESM output ingestible by PMP and (ii) the CNN psl reconstruction; neither is described reproducibly.
- "PMP-inspired" understates what was done — PMP itself is used, so "PMP-based" is more accurate. Also, all four configurations are atmosphere-only emulators with prescribed SST and SIC, so "Earth System Models" without qualification overstates their scope. Please qualify at first use.
- Ad-hoc machine learning strategy for sea-level pressure (p. 6). A residual CNN predicts psl from several inputs, but no architecture, training period, loss function, or held-out validation is given. This matters because all four modes in Section 4.2 are psl-based, so that section evaluates the DL-ESM convolved with an undocumented emulator. Please provide the full specification and an independent validation, and address the circularity: if the network is trained toward ERA5 psl, scoring it against ERA5 in Fig. 2 partly measures the emulator. Applying the same network to a CMIP6 model's surface pressure and comparing against that model's native psl would bound the effect.
The claim that the psl results are "encouraging" is also contradicted by Fig. 2, where the NeuralGCM psl cell is saturated at the top of the color scale, and Fig. 3, where both DL-ESM psl markers lie far outside the CMIP6 violin. Please revise and discuss the consequences for Section 4.2.
- Stability screening and member selection (p. 5). "Stable" is never defined. Please state the criterion and give, per model and per diagnostic, the number of usable members and the exact start and end dates:
- Figure 1. Panels (c) and (d) are both labeled "GPCP 3.2" but report different stats. Please explain and harmonize. The analysis period is annotated on only two of the six panels.
- Figure 2. Normalizing by the median of the CMIP6 models together with the DL-ESMs means the benchmark moves as models are added. Please normalize by the CMIP6-only median, or report both, so that a DL-ESM score can be read as relative to the CMIP class.
- Figure 3. Several axes extend to negative values, but RMSE is non-negative; centering each axis on the multi-model median has pushed the limits into physically impossible territory. Please clip at zero or use a logarithmic scale.
- Figure 4. The caption states that the CMIP6 results are "historical or equivalent," whereas Figs. 7 and 9 label the experiment "amip." Comparing SST-forced DL-ESMs against freely coupled historical runs for modes of variability is not like-for-like. Please state per figure which experiment was used.
- Uncertainty quantification. The manuscript only includes some acknowledgments of sampling uncertainty, and it is qualitative. There are no confidence intervals or significance tests accompanying any metric. Statements such as "NeuralGCM shows slightly superior ability to reproduce NAO" may not survive. Please add ensemble-based or bootstrap intervals, and soften comparisons that do not.
- Discussion of model errors. Explanations are offered as alternatives rather than tested: NeuralGCM-evap's precipitation errors are attributed variously to the shorter stable analysis period, a shorter training period, extrapolation, and SST sensitivity, with no attempt to discriminate among them. I would encourage the authors to exploit this further. If they prefer to keep the scope purely diagnostic, which is defensible, please moderate the claims in Section 5 about guiding model development and delivering process-specific benchmarks.
Minor Comments
- Please make the abstract quantitative: state that the configurations are atmosphere-only, give the evaluation period, and give at least one number per finding.
- Table 1. The "Boundary Conditions" column contains only a citation and is identical in every row; please name the dataset, and consider adding the training period and the analysis period used here.
- Figure 8. The orange horizontal line is not defined in the caption.
- Figure 10. The caption describes blue stars, red dots, and green triangles, but the legend uses dots for all three categories.
- For example, "Eart System modeling" (p. 3) should read "Earth"; "among the the PMP's ETMoV metrics" (p. 11) has a duplicated word, etc. Please proofread the whole manuscript before resubmission.
- Cross-references. "Section 2" and "Sect. 3" are used alternatively.
- "DL-ESM," "AI emulator," and "AI-based model" are used for the same objects; please settle on one.
Citation: https://doi.org/10.5194/egusphere-2026-3295-RC2 -
AC2: 'Reply on RC2', Giuliana Pallotta, 10 Oct 2026
We thank the anonymous reviewers for the careful and constructive evaluation of our manuscript. The comments identify several places where the initial submission did not sufficiently distinguish diagnostic results from causal interpretation, document the post-processing and stability workflow. We addressed these points through expanded discussions, a new workflow schematic and analysis-design table and revised figures.
Responses
This manuscript applies the PCMDI Metrics Package (PMP) to four deep-learning atmospheric emulator configurations, and benchmarks them against CMIP6 models and observational references. The premise is correct that forecast-oriented scores are not sufficient to establish whether a data-driven model is fit for Earth-system purposes, and the natural way to close that gap is the diagnostic battery routinely used for CMIP-class models. The breadth of diagnostics is a genuine strength, and several findings are worth publishing.
Comment 2.1
My concerns are as follows: Several methodological choices are not documented in enough detail to be reproduced or assessed, most importantly the CNN-based ad-hoc machine learning method, which underpins all of Section 4.2, and the stability screening. The comparison against CMIP6 is worth refinement. And no uncertainty is reported anywhere, although the ensembles range from ten members to one.
Response: We thank the reviewer for recognizing the value of the diagnostic breadth and agree with the three principal concerns. The revision documents the DL-ESM-to-PMP workflow (Sect. 3.2 and Fig. A1) and the sea-level-pressure reconstruction (Sect. 3.3); defines stability and diagnostic-specific member selection operationally (Sect. 2.3 and Table C1); and identifies CMIP6 AMIP as the experiment-consistent benchmark (Sect. 3.1). Claims that were not robust have been revised.
Note on referencing: the preprint carries no line numbers, so comments below are keyed to page, figure, or section number.
Major Comments
Comment 2.2
Scope and novelty. The paper applies an already-published package (Lee et al., 2024, GMD) to four existing models, so the technical novelty needs to be stated and delivered. On my reading, the new elements are (i) the workflow that makes DL-ESM output ingestible by PMP and (ii) the CNN psl reconstruction; neither is described reproducibly.
Response: We agree that the novelty must be stated more precisely. The study does not introduce a new core PMP metric. Its methodological contribution is the reproducible workflow required to evaluate heterogeneous DL-ESM output with a CMIP-class package: mapping model-specific output to CMOR-like variable conventions; temporal aggregation and calendar handling; common-grid regridding; interpolation and masking on pressure levels; missing-variable handling and sea-level-pressure reconstruction; stability screening; ensemble/member metadata; diagnostic-specific period selection; and a standardized PMP configuration for direct comparison with observations and CMIP6. The residual CNN is a separate post-processing component which we developed to be able to address the evaluated model limitations. However, we recognize that its contribution must be validated and bounded in a separate analysis which is out of the scope of the current paper.
Manuscript revision: We added the Methods subsection “DL-ESM interoperability and PMP integration” (Sect. 3.2, lines 230–244). It lists the adaptation steps (model-specific readers; CF/CMOR mapping; calendar handling and temporal aggregation; vertical interpolation and masking; psl reconstruction; stability and member screening; regridding; period selection; and the standardized PMP configuration) and states that steps (i)–(viii) are external to PMP, whereas step (ix) uses PMP version 3.9.1 as released (line 242). The workflow schematic is Fig. A1 (Appendix A), and the introduction states the contribution as an interoperability and evaluation framework rather than a new PMP metric set (lines 95–107).
Comment 2.3
"PMP-inspired" understates what was done — PMP itself is used, so "PMP-based" is more accurate. Also, all four configurations are atmosphere-only emulators with prescribed SST and SIC, so "Earth System Models" without qualification overstates their scope. Please qualify at first use.
Response: We agree on both points. However, we feel that “PMP-inspired” conveys the potential for further development of “ad hoc” PMP metrics designed specifically for evaluating DL-ESMs (work in progress), and we have kept it, with this rationale now stated in the introduction. We now qualify the evaluated systems at first use as atmosphere-only DL-ESM configurations run with prescribed sea-surface temperature and sea-ice concentration, and the term DL-ESM is used as a model-family label with the scope limitation stated in the abstract, the introduction, and the conclusions.
Manuscript revision: The title is now “A PMP-inspired evaluation framework for atmosphere-only deep-learning Earth system model configurations.” The abstract identifies the four configurations as atmosphere-only and forced by prescribed SST and sea-ice concentration (line 3); the introduction defines DL-ESM as atmosphere-only emulator configurations, states the terminology used (lines 85–89), and explains the “PMP-inspired” framing (line 105).
Comment 2.4
Ad-hoc machine learning strategy for sea-level pressure (p. 6). A residual CNN predicts psl from several inputs, but no architecture, training period, loss function, or held-out validation is given. This matters because all four modes in Section 4.2 are psl-based, so that section evaluates the DL-ESM convolved with an undocumented emulator. Please provide the full specification and an independent validation, and address the circularity: if the network is trained toward ERA5 psl, scoring it against ERA5 in Fig. 2 partly measures the emulator. Applying the same network to a CMIP6 model's surface pressure and comparing against that model's native psl would bound the effect.
The claim that the psl results are "encouraging" is also contradicted by Fig. 2, where the NeuralGCM psl cell is saturated at the top of the color scale, and Fig. 3, where both DL-ESM psl markers lie far outside the CMIP6 violin. Please revise and discuss the consequences for Section 4.2.
Response: We agree that the sea-level-pressure reconstruction was under-documented and that its errors are convolved with every pressure-based mode-of-variability result. The revised manuscript describes the reconstruction in a dedicated subsection, states that the network is trained on monthly ERA5-derived fields and uses topography, regridded to the model grid, as an input channel, and explicitly distinguishes evaluation of the reconstructed psl field from evaluation of the underlying DL-ESM: the pressure-based results are interpreted as the performance of the combined DL-ESM plus psl-reconstruction pipeline.
Manuscript revision: We removed “encouraging”. Sect. 3.3 (lines 246–251) describes the residual CNN, its predictors, its training on monthly ERA5-derived fields (line 250), and its identical application to ACE2 and NeuralGCM. Sect. 4.1 now states that “Sea-level pressure is the weakest reconstructed field for the DL-ESMs and lies outside much of the CMIP6 error distribution. Consequently, the pressure-based teleconnection results should be interpreted as performance of the combined DL-ESM plus psl-reconstruction pipeline, not of the native DL-ESM alone” (lines 323–326), and the same caveat opens Sect. 4.2 (lines 338–340). Table 2 identifies psl as a reconstructed, non-native variable.
Comment 2.5
Stability screening and member selection (p. 5). "Stable" is never defined. Please state the criterion and give, per model and per diagnostic, the number of usable members and the exact start and end dates:
Response: We agree that “stable” must be operational rather than qualitative. We now define a stable member as one that reaches the diagnostic-specific required end date without model termination, or triggering a documented physical or numerical validity check. The “end-date threshold” refers to a calendar completion date, not to a precipitation threshold; for the screening described in the manuscript, that date is 31 December 2014.
Manuscript revision: Sect. 2.3 (lines 166–181) now gives the operational definition of a stable member; states that the end date is a calendar completion date (31 December 2014 for the longest diagnostics); reports that all ten ACE2 and all ten NeuralGCM members, three of the ten NeuralGCM-evap members, and none of the twenty NeuralGCM-precip members completed that window; and explains that NeuralGCM-precip members that completed shorter windows enter only the diagnostics whose record-length requirement they satisfy. Table 1 gives the ensemble size and initialization of each configuration, and Table C1 the members entering each diagnostic (e.g., eight NeuralGCM-evap and six NeuralGCM-precip members for the MJO ensemble statistics).
Comment 2.6
Figure 1. Panels (c) and (d) are both labeled "GPCP 3.2" but report different stats. Please explain and harmonize. The analysis period is annotated on only two of the six panels.
Response: We agree. Both panels used GPCP 3.2 but corresponded to different model-specific averaging windows. We regenerated the figure over a single common period (January 1980 to December 2014) with the more appropriate reference dataset GPCP 2.3, so the repeated observational panels now show the same reference climatology.
Manuscript revision: Fig. 1 has been regenerated over the common period with the reference GPCP 2.3 dataset, and its caption states the period and the reference product.
Comment 2.7
Figure 2. Normalizing by the median of the CMIP6 models together with the DL-ESMs means the benchmark moves as models are added. Please normalize by the CMIP6-only median, or report both, so that a DL-ESM score can be read as relative to the CMIP class.
Response: We agree. The benchmark should not move when a DL-ESM is added. Fig. 2 shows the normalized spatial RMSE with respect to the CMIP6 models only, and its caption now describes the normalized errors relative to the CMIP6 median.
Manuscript revision: Sect. 4.1 (lines 289–291) describes the normalized spatial RMSE shown in the portrait plots, and the Fig. 2 caption states that negative (positive) values indicate lower (higher) RMSE than the CMIP6 median.
Comment 2.8
Figure 3. Several axes extend to negative values, but RMSE is non-negative; centering each axis on the multi-model median has pushed the limits into physically impossible territory. Please clip at zero or use a logarithmic scale.
Response: We agree that negative RMSE axis ranges are physically meaningless. They arose from plotting offsets around a central value rather than from negative RMSE values. The figure has been replotted with axes that start at zero.
Manuscript revision: Fig. 3 has been replotted with the vertical axes clipped at zero, and its caption describes the spatiotemporal RMSE of the climatological annual cycle (Sect. 4.1, line 312).
Comment 2.9
Figure 4. The caption states that the CMIP6 results are "historical or equivalent," whereas Figs. 7 and 9 label the experiment "amip." Comparing SST-forced DL-ESMs against freely coupled historical runs for modes of variability is not like-for-like. Please state per figure which experiment was used.
Response: We agree that the comparison experiment must be stated and matched. The DL-ESMs are SST-forced atmosphere-only simulations, so CMIP6 AMIP is the primary like-for-like benchmark. Coupled historical simulations include internally generated SST variability and should not be pooled with AMIP without explicit qualification. The revised manuscript states this explicitly.
Manuscript revision: Sect. 3.1 (lines 212–213 and 222–228) now states that CMIP6 AMIP is the primary comparator for these SST-forced atmosphere-only runs and that coupled historical simulations are not pooled with AMIP results. The Figs. 4 and 5 captions state that their CMIP6 results come from AMIP simulations.
Comment 2.10
Uncertainty quantification. The manuscript only includes some acknowledgments of sampling uncertainty, and it is qualitative. There are no confidence intervals or significance tests accompanying any metric. Statements such as "NeuralGCM shows slightly superior ability to reproduce NAO" may not survive. Please add ensemble-based or bootstrap intervals, and soften comparisons that do not.
Response: We agree that quantifying sampling uncertainty with a hierarchical resampling strategy would be meaningful for the key climatology, teleconnection, MJO, monsoon, and spectral metrics. However this analysis, although useful, goes beyond the scope of this first evaluation.
Manuscript revision: To account for the lack of quantitative uncertainty quantification, statements such as “slightly superior” have been removed (e.g., the NAO discussion in Sect. 4.2, lines 361–372).
Comment 2.11
Discussion of model errors. Explanations are offered as alternatives rather than tested: NeuralGCM-evap's precipitation errors are attributed variously to the shorter stable analysis period, a shorter training period, extrapolation, and SST sensitivity, with no attempt to discriminate among them. I would encourage the authors to exploit this further. If they prefer to keep the scope purely diagnostic, which is defensible, please moderate the claims in Section 5 about guiding model development and delivering process-specific benchmarks.
Response: We agree that the original discussion listed possible explanations without discriminating among them. The revised Discussion relates each result to the relevant model-design difference and presents training-period length, extrapolation, SST sensitivity, and architecture as hypotheses rather than established causes. Fig. 1 has been regenerated over a common period, which removes the period mismatch between ACE2 and NeuralGCM-evap in the precipitation-climatology comparison.
Manuscript revision: The opening paragraph of Sect. 5 states that the framework “identifies where model behavior diverges and thereby prioritizes hypotheses and follow-up experiments” (lines 504–512); Sect. 5.1 presents the mechanisms behind each result as hypotheses (lines 514–539); the NeuralGCM-evap explanations in Sect. 4.1 are presented as possible contributors (lines 280–286 and 296–302); and the final sentence of the paper refers to “process-relevant diagnostics that help prioritize model development” (line 582).
Minor Comments Comment 2.12
Please make the abstract quantitative: state that the configurations are atmosphere-only, give the evaluation period, and give at least one number per finding.
Response: Agreed. The abstract now identifies the configurations as atmosphere-only and gives the common 1981–2013 climatology period and the shorter diagnostic-dependent periods, and reports representative quantitative results.
Manuscript revision: The abstract (lines 2–14) states that the four configurations are atmosphere-only and SST/SIC-forced; gives the common 1981–2013 climatology period and the shorter diagnostic-dependent periods; and states that ACE2 and NeuralGCM have climatological errors below the CMIP6 median for pressure-level temperature and wind, whereas the errors of their reconstructed sea-level pressure lie outside the CMIP6 range. It also reports the annual-mean precipitation pattern correlations of ACE2 (0.93) and NeuralGCM-evap (0.63) and that ACE2 captures the pattern of annual precipitation variability (correlation 0.84) but overestimates its amplitude by about 56 %.
Comment 2.13
Table 1. The "Boundary Conditions" column contains only a citation and is identical in every row; please name the dataset, and consider adding the training period and the analysis period used here.
Response: Agreed. Table 1 now gives, for each configuration, the reference, model type, precipitation and evaporation treatment, deterministic or stochastic mode, training data and period, resolution, and ensemble construction; its footnote names the PCMDI-AMIP SST and sea-ice boundary conditions, whose version (v1.1.9) is given in Sect. 3.1. The analysis periods and members are listed per figure in Table C1.
Comment 2.14
Figure 8. The orange horizontal line is not defined in the caption.
Response: Agreed. The Fig. 8 caption now defines the orange horizontal line as the CMIP6 multi-model mean EWR.
Comment 2.15
Figure 10. The caption describes blue stars, red dots, and green triangles, but the legend uses dots for all three categories.
Response: Agreed. The Fig. 10 caption now matches the legend and the plotted markers (blue, red, and green dots for hits, misses, and false alarms).
Comment 2.16
For example, "Eart System modeling" (p. 3) should read "Earth"; "among the the PMP's ETMoV metrics" (p. 11) has a duplicated word, etc. Please proofread the whole manuscript before resubmission.
Response: Agreed. We corrected the cited errors (“Eart” to “Earth” and the duplicated “the”) and carried out a proofreading pass covering punctuation, capitalization, units, and figure cross-references.
Comment 2.17
Cross-references. "Section 2" and "Sect. 3" are used alternatively.
Response: Agreed. “Section” is now used only at the start of a sentence and “Sect.” elsewhere, following the Copernicus style, and the same rule is applied to “Figure” and “Fig.”.
Comment 2.18
"DL-ESM," "AI emulator," and "AI-based model" are used for the same objects; please settle on one.
Response: Agreed. We now use “atmosphere-only DL-ESM configuration” (or DL-ESM) as the primary term and “AI-based weather model” only when referring to the broader literature; this terminology statement appears at first use in the introduction (lines 85–89).
Citation: https://doi.org/10.5194/egusphere-2026-3295-AC2
-
RC3: 'Comment on egusphere-2026-3295', Anonymous Referee #3, 31 Jul 2026
This manuscript presents a realistic and timely framework for comparing and validating the performance of different Deep Learning Earth System Models (DL-ESMs), which provides a very useful contribution to the Earth system modelling community. The manuscript is generally well written and clearly structured. However, there are a few key points and technical details that could be further clarified or expanded upon to strengthen the manuscript.
As the paper is submitted to GMD, it would be beneficial for the authors to further highlight the model development aspects of this work. Given that the core evaluation package (PMP) has been published previously, could the authors better articulate what new developments were implemented in PMP for this study, and what new developments were in the pre- and post-processing of DL-ESM outputs? At present, the manuscript reads more like an application of PMP rather than a development framework, and clarifying this distinction would help align it firmly with GMD's scope.
Additionally, a few general aspects regarding the core content could be improved:
- Explanations for some key metrics are missing. Furthermore, the visual presentation of some figures could be enhanced (e.g., increasing font sizes for readability and reducing large blank spaces without useful information).
- The rationale behind selecting specific realizations and study periods throughout the paper is not always clear. While the authors note "exposing model-specific limitations such as missing output variables, post-processing requirements, and stability constraints", it would be clearer to explicitly outline the criteria for choosing these different periods and realizations.
- While ta-200, ta-850, va-200, va-850, zg-200, and zg-850 are listed as key variables, there is limited discussion of them in the main text.
- The “stability analysis” seems to be missing.
- Adding a schematic diagram showing the complete workflow—including pre-processing, post-processing of DL-ESM outputs, and the integration with PMP—would greatly assist readers in visualizing the overall framework.
- Overall, deepening the discussion of the results in a few key sections would help maximize the paper's impact.
Major revisions
- Page 4, second paragraph. Regarding the statement, “These models differ both in architecture and in the way they represent atmospheric evolution,” while the structural differences between ACE2 and NeuralGCM are clear, could the authors briefly elaborate on the specific structural differences among the three NeuralGCM variants?
- Page 4, last paragraph. Could you please clarify whether the random perturbations are applied to all model parameters or only a subset? Additionally, specifying their statistical distributions (e.g., Gaussian vs. uniform) could be very useful.
- Page 5, first paragraph. Why the way of generating initial condition ensemble for ACE2 and NeuralGCM are different? The first one is perturbated by choosing between different days and the latter is by choosing from different months.
- Page 5, second paragraph. Could the authors please clarify how "long-run" stability is defined and what the "end-date threshold" refers to (e.g., is it a specific precipitation threshold)? Rewording this paragraph would clear up any potential confusion.
- Table 2. Sea level pressure is listed as one of the variables to be evaluated, while in Page 2 second paragraph, the “global mean log logarithm of surface pressure” is fixed. What’s the impact of this correction on the evaluation? And what’s the main purpose of this correction?
- Figure 1. It is not clear how the RMSE is calculated (e.g., across all grid cells over time versus time series of global averages). In Figure 1c, the longitudinal tick labels appear partially cut off, and the time period for the map seems to be missing.
- Page 8. Regarding the sentence “while the remaining RMSE (0.80 mm day−1) highlights persistent tropical amplitude errors”, could the authors clarify how the "persistent" is defined or demonstrated here?
- Page 10, line 8. The “lack of stability” needs to be explained.
- Page 10. Could the authors explain the “spatiotemporal RMSE”, and how it is different from other RMSE calculations.
- Figure 4. What criteria were used to select the specific ensemble members presented in this calculation?
- Figure 5. For SAM_MAM, NAO_DJF, and PNA_JJA, both two DL-ESMs show quite large biases compared to the medians, and the directions of the biases are consistent for these two DL-ESMs. Could the authors offer some insight or hypotheses as to what might be driving these consistent biases?
- Figures 5, 6 and 9. Do the r1i1p1f1, r53i1p1f1, and r100i1p1f1 have the same meaning as in CMIP data naming conventions? Could you explain why these specific realizations were selected for display, and why the f tag is omitted in Figure 9?
- Page 14, last paragraph. The differences in EWR among three DL-ESMs and obs are compared here. However, based on Figure 7, none of the three DL-ESMs reproduce the dominance of wavenumber 1 and 3 appeared in the observations. Rather, all of them show the dominance of wavenumber 2. It would be great if the discussion could address this wavenumber shift alongside the EWR values. Also, based on EWR, NeuralGCM-evap better replicates the obs. But between ACE2 and NeuralGCM-precip, which one is better? Why were different time periods selected for the DL-ESMs versus the obs in this analysis?
- Page 16, first paragraph. Regarding the sentence “These comparisons can be potentially affected by the ensemble-size differences, because the DL-ESM ensembles contain 10 members, whereas, several CMIP6 models contribute to the analysis with fewer members”, it would be helpful to discuss how sensitive the DL-ESMs' mean EWR is to the chosen ensemble size.
- Figure 8. In terms of May-October EWR, the three DL-ESMs appear to outperform some CMIP6 models. It would be helpful to have some figures of the wavenumber-frequency power spectra for CMIP6 models.
- Page17. In the sentence “They also produce more widespread false alarms across the tropical oceans”, could the authors provide quantitative counts for these false alarms to complement the qualitative description?
- Figures 9 and 10. Could you please clarify why different study periods were selected for Figure 9 versus Figure 10?
- Page 20. Regarding the sentence “However, the NeuralGCM-precip precipitation/convection formulation appears to improve behavior at synoptic timescales relative to longer-period variability”, could the authors present additional visual evidence (such as spectral power ratios across frequencies) to further back up this?
- Page 20. In sentence “an apparently reasonable global amplitude can mask poor geographic placement”, could you elaborate on what defines a "reasonable" amplitude here? Does ACE2's amplitude contribute significantly to its high spatial correlation, and does this suggest that visual inspection of spatial maps remains necessary alongside summary metrics?
- Page 21. In the sentence “NeuralGCM is comparable to the best-performing CMIP6 models, whereas ACE2 lies closer to the lower-skill edge of the CMIP6 distribution for this field”. Could you explain what is this lower-skill edge?
- Page 23. In the sentence “NeuralGCM-evap and NeuralGCM-precip show greater sensitivity to initial conditions, random seeds, and run length”, where is the discussion to support this?
- Page 23, third paragraph. Please ensure that appropriate literature references are added to complete this paragraph.
Minor revisions
- Page 2 line 28. It would be helpful to include a few concrete examples of standard weather-forecast scores, along with a brief note on their respective strengths and limitations.
- Page 3 line 24. Typo: please change to "Earth System Modeling".
- Page 9, line 11. A summary table for all the annual and seasonal climatology fields could be helpful. Additionally, increasing the font size in the Supplementary Figures would improve readability.
- Page 9, line 13. Could you provide an explicit formula or example showing how the global spatial RMSE is calculated?
- Figure 2. Please consider increasing the font size for clarity. Rotating the matrix so that models are on the y-axis might also make the figure easier to read.
- Page 10 line 10. In the sentence “We also notice the good performance of ACE2 across all the output variables”, the performance of ta-200 and ta-850 appears somewhat weaker; the authors may wish to nuance this sentence slightly.
- Page 10 line 11. As "Root Mean Square Error" has already been introduced, the acronym "RMSE" can be used directly here.
- Figure 6. Could you please clarify the precise definition of "NAO skill"? It would also be helpful to discuss why the values simulated by both DL-ESMs over the North Pacific are notably higher.
- Figure 11. Adjusting the y-axis limits to zoom in on the active data range (as there are currently no data above 1 or below 10-3) would enhance visual clarity.
- Figure 13. Consider using insets or re-scaling the plot area to reduce unused white space and highlight the data.
Citation: https://doi.org/10.5194/egusphere-2026-3295-RC3 -
AC3: 'Reply on RC3', Giuliana Pallotta, 10 Oct 2026
We thank the anonymous reviewers for the careful and constructive evaluation of our manuscript. The comments identify several places where the initial submission did not sufficiently distinguish diagnostic results from causal interpretation, document the post-processing and stability workflow. We addressed these points through expanded discussions, a new workflow schematic and analysis-design table and revised figures.
Reviewer 3
This manuscript presents a realistic and timely framework for comparing and validating the performance of different Deep Learning Earth System Models (DL-ESMs), which provides a very useful contribution to the Earth system modelling community. The manuscript is generally well written and clearly structured. However, there are a few key points and technical details that could be further clarified or expanded upon to strengthen the manuscript.
As the paper is submitted to GMD, it would be beneficial for the authors to further highlight the model development aspects of this work. Given that the core evaluation package (PMP) has been published previously, could the authors better articulate what new developments were implemented in PMP for this study, and what new developments were in the pre- and post-processing of DL-ESM outputs? At present, the manuscript reads more like an application of PMP rather than a development framework, and clarifying this distinction would help align it firmly with GMD's scope.
Response: We thank the reviewer for this helpful framing. We agree that the manuscript must distinguish use of an existing diagnostic package from the development needed to make heterogeneous DL-ESM output evaluable by that package. No new PMP metric is claimed. The development contribution is the interoperable preprocessing, post-processing, stability, metadata, and diagnostic-orchestration workflow, together with the psl-reconstruction component. The revised manuscript makes this distinction explicit in the introduction and in the new Appendix A, B and C, and it documents the workflow as a reproducible model-evaluation capability.
Manuscript revision: The new Sect. 3.2 (lines 230–244) documents the development steps: model-specific readers; CMOR variable mapping; calendar handling and temporal aggregation; pressure interpolation and masking; missing-variable handling and psl reconstruction; stability and member screening; regridding; diagnostic-specific period selection; and the PMP configuration. It states that steps (i)–(viii) are external pre- and post-processing and that step (ix) uses PMP version 3.9.1 as released. The newly added schematic is Fig. A1 (Appendix A).
Comment 3.1
Additionally, a few general aspects regarding the core content could be improved:
Explanations for some key metrics are missing. Furthermore, the visual presentation of some figures could be enhanced (e.g., increasing font sizes for readability and reducing large blank spaces without useful information).
Response: Agreed. The revised manuscript defines each metric where it is first used (the normalized spatial RMSE and the spatiotemporal RMSE in Sect. 4.1; the CBF amplitude ratio, pattern RMSE, and explained variance in Sect. 4.2; the MJO power ratios in Sect. 4.3; the monsoon skill metrics in Sect. 4.4; and the spectral power ratios and timescale bands in Sect. 4.5), refers to Lee et al. (2024) for the formal PMP definitions.
Manuscript revision: We added Table B1 (Appendix B) which summarizes each diagnostic, the model property it tests, the variables and the reference data it requires. It is referenced from the introduction (lines 95–107) and from Sects. 4.4 and 4.5 (lines 426–438 and 449–451).
Comment 3.2
The rationale behind selecting specific realizations and study periods throughout the paper is not always clear. While the authors note "exposing model-specific limitations such as missing output variables, post-processing requirements, and stability constraints", it would be clearer to explicitly outline the criteria for choosing these different periods and realizations.
Response: Agreed. Periods and realizations were chosen from the intersection of available variables, numerical completion, observational coverage, and the minimum record length required by each diagnostic. The revised manuscript makes this logic explicit and distinguishes numerical stability and diagnostic record-length requirements (Sect. 2.3) from data availability (Sect. 2.4 and Table 2).
Manuscript revision: Table C1 (Appendix C) lists, for every figure, the DL-ESM configurations and members, the DL-ESM periods, and the reference dataset and period; it is referenced from Sects. 2.3, 3.1, and 4 and from the figure captions. Sect. 2.3 (lines 166–181) defines numerical stability and explains how record length determines which members enter each diagnostic, and Sect. 2.4 (lines 192–199) identifies missing variables as an availability limitation rather than performance-based selection.
Comment 3.3
While ta-200, ta-850, va-200, va-850, zg-200, and zg-850 are listed as key variables, there is limited discussion of them in the main text.
Response: Agreed. Sect. 4.1 now includes a synthesis of the pressure-level fields that compares the temperature and wind errors at 200 and 850 hPa among the DL-ESMs and against the CMIP6 distribution and relates the low 200 hPa wind errors to the representation of the subtropical jets and stationary waves.
Manuscript revision: Sect. 4.1 no longer describes ACE2 as uniformly strong: it adds a paragraph on the pressure-level fields (lines 317–326) that identifies the fields for which the DL-ESM errors lie at or below the lower edge of the CMIP6 distribution (e.g., NeuralGCM ta-850 and ua-850) and those, such as the reconstructed sea-level pressure, that lie outside it.
Comment 3.4
The “stability analysis” seems to be missing.
Response: Agreed. A dedicated subsection (Sect. 2.3) now defines the stability tests, reports which members completed the required windows, and explains how stability constraints alter the sample entering each diagnostic (Table C1).
Manuscript revision: Sect. 2.3 (lines 166–181) defines numerical stability (completion of the required window without model termination, non-finite output, or validity-check failure), reports the completion counts for the window ending on 31 December 2014, and explains which diagnostics the shorter integrations enter; Sect. 2.4 treats missing variables separately, as an availability limitation.
Comment 3.5
Adding a schematic diagram showing the complete workflow—including pre-processing, post-processing of DL-ESM outputs, and the integration with PMP—would greatly assist readers in visualizing the overall framework.
Response: Agreed. We added a workflow schematic (Fig. A1, Appendix A) showing the initial and boundary conditions; the native model output and metadata; the DL-ESM interoperability steps (model-specific readers; variable, unit, and calendar harmonization; pressure-level interpolation and masking; psl reconstruction where required; stability and member screening; regridding); the standard PMP diagnostics and the comparison with observations and CMIP6. The schematic distinguishes the components developed in this study from the standard PMP diagnostics.
Comment 3.6
Overall, deepening the discussion of the results in a few key sections would help maximize the paper's impact.
Response: Agreed. The new Sect. 5.1 (lines 514–539) deepens the discussion of the precipitation climatology, the extratropical modes of variability, the MJO, the monsoons, and precipitation variability across timescales. For each, it states what the metric demonstrates, the relevant architecture and training differences, and plausible mechanisms, phrased as hypotheses. The subsequent newly added Sections 5.2 and 5.3 help clarify the impact.
Major revisions Comment 3.7
Page 4, second paragraph. Regarding the statement, “These models differ both in architecture and in the way they represent atmospheric evolution,” while the structural differences between ACE2 and NeuralGCM are clear, could the authors briefly elaborate on the specific structural differences among the three NeuralGCM variants?
Response: We expanded this description. The original NeuralGCM couples a differentiable atmospheric dynamical core to learned physical parameterizations. NeuralGCM-evap and NeuralGCM-precip retain the hybrid dynamical-core framework but are precipitation-focused stochastic configurations with different hydrologic closure: the evap configuration predicts evaporation and diagnoses precipitation from the atmospheric water budget, whereas the precip configuration predicts precipitation directly and diagnoses evaporation. They also differ from the original checkpoint in training data and period and in stochastic inference, so they should not be interpreted as a single-factor ablation.
Manuscript revision: Sect. 2.1 (lines 137–148) now describes the three NeuralGCM configurations, their hydrologic closure, and the training data (ERA5 and IMERG V07 precipitation) and period (2001–2018) of the precipitation-focused variants. Table 1 compares the configurations by model type, precipitation and evaporation treatment, stochastic components, training data and period, resolution, and ensemble design.
Comment 3.8
Page 4, last paragraph. Could you please clarify whether the random perturbations are applied to all model parameters or only a subset? Additionally, specifying their statistical distributions (e.g., Gaussian vs. uniform) could be very useful.
Response: The random perturbations are not applied to trained model parameters. Initial-condition ensembles use different initial atmospheric states. In the stochastic NeuralGCM variants, changing the random seed generates a different realization of the stochastic fields used within the learned parameterization, and the trained weights remain fixed. The revised Sects. 2.1 and 2.2 state this explicitly.
Manuscript revision: Sect. 2.2 (line 160) states that each seed generates a different realization of these fields and that no parameter-space perturbation is applied. The ambiguous phrase “random perturbations are introduced into the embedded physical parameterizations” has been removed.
Comment 3.9
Page 5, first paragraph. Why the way of generating initial condition ensemble for ACE2 and NeuralGCM are different? The first one is perturbated by choosing between different days and the latter is by choosing from different months.
Response: The start-date choices followed the valid initialization protocol and available initial-state archive for each released checkpoint, rather than an intent to give ACE2 and NeuralGCM different sampling. ACE2 members were selected from available daily initial states, whereas the deterministic NeuralGCM ensemble used one initialization per month on the documented day. We agree that this creates a potential calendar-sampling confound, and the revised Sect. 2.2 states that the two ensembles are therefore not strictly exchangeable.
Manuscript revision: Sect. 2.2 (lines 155–164) gives the selection rule for ACE2 (start dates offset by one or more days) and NeuralGCM (the 15th of each month from February to November 1979) and states that, because the two ensembles sample different initialization calendars (daily versus monthly offsets), they should not be interpreted as strictly exchangeable. Table 1 adds the design of the stochastic ensembles (a December 1979 start date with ten and twenty random seeds for NeuralGCM-evap and NeuralGCM-precip, respectively).
Comment 3.10
Page 5, second paragraph. Could the authors please clarify how "long-run" stability is defined and what the "end-date threshold" refers to (e.g., is it a specific precipitation threshold)? Rewording this paragraph would clear up any potential confusion.
Response: We agree that this paragraph was ambiguous. “Long-run stability” is now defined as completing the required simulation window with finite fields and without triggering an explicitly listed numerical or physical validity criterion. The “end-date threshold” is the required calendar end date (31 December 2014 in the stated screening), not a precipitation or other field-value threshold.
Manuscript revision: The paragraph has been rewritten in Sect. 2.3 (lines 166–181) to report the completion counts, the failure criteria, the use of the correction, and the diagnostic-specific inclusion of members, and “lack of stability” in Sect. 4.1 has been replaced with the observed fact that none of the NeuralGCM-precip members completed the 1981–2013 climatology period (line 300).
Comment 3.11
Table 2. Sea level pressure is listed as one of the variables to be evaluated, while in Page 2 second paragraph, the “global mean log logarithm of surface pressure” is fixed. What’s the impact of this correction on the evaluation? And what’s the main purpose of this correction?
Response: The fixed global mean of log surface pressure is a mass-conservation adjustment applied during NeuralGCM rollout to prevent drift in globally integrated atmospheric mass. It changes only the global-mean component of log surface pressure at each step; it is not a direct correction of local sea-level-pressure patterns. Nevertheless, local surface pressure is an input to the residual CNN used to infer psl, so the correction can propagate into the derived field.
Manuscript revision: Sect. 2.3 (line 171) explains the purpose of the correction, states that it changes only the global-mean component of log surface pressure and is neither a correction of local pressure patterns nor a statistical bias correction, and notes that it can propagate into the reconstructed psl because surface pressure is an input to the reconstruction.
Comment 3.12
Figure 1. It is not clear how the RMSE is calculated (e.g., across all grid cells over time versus time series of global averages). In Figure 1c, the longitudinal tick labels appear partially cut off, and the time period for the map seems to be missing.
Response: The map RMSE is the area-weighted spatial RMSE between model and reference climatological fields on a common grid, not the RMSE of global-mean time series. This definition is now stated in the Fig. 1 caption.
Manuscript revision: The caption of the regenerated Fig. 1 states that the bias panels are annotated with the area-weighted spatial RMSE and the spatial pattern correlation, and it gives the common averaging period.
Comment 3.13
Page 8. Regarding the sentence “while the remaining RMSE (0.80 mm day−1) highlights persistent tropical amplitude errors”, could the authors clarify how the "persistent" is defined or demonstrated here?
Response: We agree that “persistent” was not demonstrated by a single climatological RMSE and replaced it with wording that refers to systematic errors in the time-mean climatology.
Manuscript revision: The sentence now reads: “The high spatial correlation (0.93) indicates that the large-scale pattern is realistic, while the remaining RMSE of 0.80 mm day−1 reflects systematic tropical amplitude and placement errors in the evaluated climatology” (line 273).
Comment 3.14
Page 10, line 8. The “lack of stability” needs to be explained.
Response: Agreed. We replaced “lack of stability” with the specific condition that limits the analysis: none of the NeuralGCM-precip members completed the 1981–2013 climatology period, so this configuration enters only the diagnostics that require shorter records (Sect. 4.1, line 300; Sect. 2.3).
Comment 3.15
Page 10. Could the authors explain the “spatiotemporal RMSE”, and how it is different from other RMSE calculations.
Response: In this manuscript, the spatiotemporal RMSE summarizes the climatological annual cycle rather than a single annual-mean map. For each calendar month, PMP computes the area-weighted spatial RMSE between the model and reference monthly climatologies and then averages the 12 monthly values with weights proportional to month length, so the square root is taken before the temporal average. This differs from the spatial RMSE of the annual-mean field because it retains seasonal-cycle errors.
Manuscript revision: Sect. 4.1 (line 312) refers to Lee et al. (2024) for the definition of the spatiotemporal RMSE, and the Fig. 3 caption describes it as the error of the climatological annual cycle, distinguishing it from the spatial RMSE of the seasonal climatologies in Figs. 1 and 2.
Comment 3.16
Figure 4. What criteria were used to select the specific ensemble members presented in this calculation?
Response: No member was selected based on favorable metric performance. Figure 4 uses all members that satisfy the pressure-variable availability, stability, record-length, and common-period criteria for that calculation. The ensemble statistic is then computed consistently across those usable members.
Manuscript revision: The Fig. 4 caption now states that all usable members are included (ten each for ACE2 and NeuralGCM), that no member was selected on the basis of performance, and that the metric per model is the average over its members; Table C1 lists the members and periods of each figure.
Comment 3.17
Figure 5. For SAM_MAM, NAO_DJF, and PNA_JJA, both two DL-ESMs show quite large biases compared to the medians, and the directions of the biases are consistent for these two DL-ESMs. Could the authors offer some insight or hypotheses as to what might be driving these consistent biases?
Response: The consistent departures in these mode–season combinations may reflect shared elements of the evaluation pipeline and forcing, including the prescribed SST and SIC, reanalysis-based training, and the common psl-reconstruction method, as well as shared circulation errors such as jet latitude or strength. Because ACE2 and NeuralGCM have different architectures, the agreement should not be attributed to one structural component without additional tests. In the figure, the shared PNA departure appears in MAM rather than JJA, so the revised text refers to SAM MAM and NAO DJF, where both DL-ESMs lie in the upper part of the CMIP6 distribution, and to PNA MAM, where both lie below the median.
Manuscript revision: Sect. 4.2 (lines 355–360) now describes these shared departures and presents their possible causes as hypotheses.
Comment 3.18
Figures 5, 6 and 9. Do the r1i1p1f1, r53i1p1f1, and r100i1p1f1 have the same meaning as in CMIP data naming conventions? Could you explain why these specific realizations were selected for display, and why the f tag is omitted in Figure 9?
Response: The labels follow the CMIP variant-label syntax: r denotes realization/member, i initialization method, p physics, and f forcing. For the DL-ESM output, i1p1f1 are bookkeeping labels held fixed, while r identifies the generated member/start-date/seed combination; they do not imply distinct CMIP-style physics or forcing variants.
Comment 3.19
Page 14, last paragraph. The differences in EWR among three DL-ESMs and obs are compared here. However, based on Figure 7, none of the three DL-ESMs reproduce the dominance of wavenumber 1 and 3 appeared in the observations. Rather, all of them show the dominance of wavenumber 2. It would be great if the discussion could address this wavenumber shift alongside the EWR values. Also, based on EWR, NeuralGCM-evap better replicates the obs. But between ACE2 and NeuralGCM-precip, which one is better? Why were different time periods selected for the DL-ESMs versus the obs in this analysis?
Response: We agree. A scalar east–west power ratio can appear close to observations while the wavenumber distribution remains wrong. The spectra show that the observations emphasize eastward wavenumbers 1 and 3, whereas the three DL-ESMs place excessive relative power at wavenumber 2; the revised text discusses this structural shift explicitly. For the displayed members, NeuralGCM-evap is closest to the observed EWR of about 2.49, and ACE2 (about 2.05) is closer than NeuralGCM-precip (about 3.52), although neither scalar comparison resolves the incorrect spectral placement. We also regenerated Fig. 8, which places the ensemble-mean EWRs in the context of CMIP6: the ACE2 ensemble mean (approximately 2.4) is close to the observed 2.49, the NeuralGCM-evap ensemble mean is somewhat higher (approximately 2.7), and the NeuralGCM-precip ensemble mean (approximately 3.3) lies well above the reference, consistent with its single member in Fig. 7. Table C1 lists the members (ten, eight, and six) and the period (1985–2004).
Manuscript revision: Sect. 4.3 (lines 386–397) now states the periods of the displayed spectra and that they differ because the daily GPCP 1.3 record begins in October 1996, notes that the observed spectrum peaks at eastward wavenumbers 1 and 3 near 45-day periods whereas the DL-ESM members concentrate eastward power near wavenumber 2 (with the ACE2 maximum at periods longer than about 60 days, outside the EWR band), and explains that the scalar EWR does not capture where the power lies (line 391). The ensemble means of the regenerated Fig. 8 are compared with the single-member values in lines 405–407.
Comment 3.20
Page 16, first paragraph. Regarding the sentence “These comparisons can be potentially affected by the ensemble-size differences, because the DL-ESM ensembles contain 10 members, whereas, several CMIP6 models contribute to the analysis with fewer members”, it would be helpful to discuss how sensitive the DL-ESMs' mean EWR is to the chosen ensemble size.
Response: We agree that quantifying ensemble-size sensitivity by repeatedly subsampling k members from each DL-ESM ensemble and recomputing the ensemble-mean EWR would be an interesting addition to the current analysis as future work.
Comment 3.21
Figure 8. In terms of May-October EWR, the three DL-ESMs appear to outperform some CMIP6 models. It would be helpful to have some figures of the wavenumber-frequency power spectra for CMIP6 models.
Response: We agree that wavenumber-frequency power spectra can help show whether a favorable scalar EWR corresponds to realistic spectral structure. However, this type of additional analysis goes beyond the scope of the current work.
Comment 3.22
Page17. In the sentence “They also produce more widespread false alarms across the tropical oceans”, could the authors provide quantitative counts for these false alarms to complement the qualitative description?
Response: Agreed. Raw grid-cell counts depend on resolution, so the revised text states that the markers in Fig. 10 are shown on each model's native grid, which gives ACE2 (1°) a denser set of markers than the 2.8° NeuralGCM variants, and notes that raw grid-cell counts therefore depend on resolution.
Comment 3.23
Figures 9 and 10. Could you please clarify why different study periods were selected for Figure 9 versus Figure 10?
Response: The periods differ because the two diagnostics use different GPCP products and record lengths (daily GPCP 1.3, available only from October 1996, for the pentad accumulations; monthly GPCP 2.3 for the annual range) and because some model members do not remain usable for the longer interval. We agree that this must be stated explicitly and that period differences can affect the comparison.
Manuscript revision: The Figs. 9 and 10 captions and Table C1 now give all periods: 1980–2005 for the models and 1997–2016 for GPCP 1.3 in Fig. 9, and 1990–2005 for the models and 1979–2019 for GPCP 2.3 in Fig. 10. Sect. 4.4 notes that the two figures use different reference products and periods (lines 426–438).
Comment 3.24
Page 20. Regarding the sentence “However, the NeuralGCM-precip precipitation/convection formulation appears to improve behavior at synoptic timescales relative to longer-period variability”, could the authors present additional visual evidence (such as spectral power ratios across frequencies) to further back up this?
Response: We agree that the original sentence was stronger than the displayed evidence and removed it. The timescale bands are now defined before the results (synoptic, 1–20 days; sub-seasonal, 20–90 days; seasonal, 90–365 days; and interannual, longer than 365 days), and the text no longer infers an improved representation of convection from the precipitation spectra.
Manuscript revision: Sect. 4.5 defines the bands following Ahn et al. (2022) (lines 449–451) and now states that “direct precipitation prediction alone does not guarantee realistic variability and precipitation spectra alone cannot be used to infer an improved representation of convection” (line 459).
Comment 3.25
Page 20. In sentence “an apparently reasonable global amplitude can mask poor geographic placement”, could you elaborate on what defines a "reasonable" amplitude here? Does ACE2's amplitude contribute significantly to its high spatial correlation, and does this suggest that visual inspection of spatial maps remains necessary alongside summary metrics?
Response: We agree that “reasonable” is subjective and replaced it with the numerical amplitude ratio. A spatial pattern correlation is insensitive to uniform amplitude scaling, so the amplitude of ACE2 does not mechanically produce its high correlation. In fact, the amplitude ratio of about 1.56 indicates a substantial positive amplitude bias even where the spatial pattern correlation is high, which is why maps, amplitude metrics, and correlation must be interpreted together.
Manuscript revision: The quoted phrase now reads “an amplitude ratio near one can coexist with poor geographic placement”, followed by “ACE2 reproduces the broad spatial pattern but overestimates global variability amplitude by approximately 56 % in this metric; the summary correlation therefore must be interpreted together with accurate amplitude and regional placement” (lines 470–473).
Comment 3.26
Page 21. In the sentence “NeuralGCM is comparable to the best-performing CMIP6 models, whereas ACE2 lies closer to the lower-skill edge of the CMIP6 distribution for this field”. Could you explain what is this lower-skill edge?
Response: We agree that “lower-skill edge” was vague; it was also inconsistent with the figure, in which ACE2 is the DL-ESM closest to the ERA5 reference for 850 hPa temperature. We replaced the sentence with a description based on the plotted normalized standard deviations and pattern correlations.
Manuscript revision: Sect. 4.6 (lines 483–486) now states that ACE2 lies very close to the ERA5 reference (normalized standard deviation of approximately 1.03 and an almost perfect pattern correlation), that NeuralGCM has a similarly high pattern correlation but overestimates the spatial standard deviation (approximately 1.09), and that both lie within the best-performing part of the CMIP6 distribution. The descriptions of the precipitation and 201 hPa wind panels were also corrected: the normalized standard deviation of NeuralGCM-evap is approximately 1.4 (line 491), and both DL-ESMs have approximately 0.92 for ua-200 (line 497). The Fig. 13 caption now states that the CMIP6 comparison uses AMIP simulations for 1981–2013.
Comment 3.27
Page 23. In the sentence “NeuralGCM-evap and NeuralGCM-precip show greater sensitivity to initial conditions, random seeds, and run length”, where is the discussion to support this?
Response: We agree that this synthesis must be supported or removed, and we replaced it with a narrower statement: the revised text no longer claims that the precipitation-focused variants are more sensitive to initial conditions, random seeds, or run length, and the completion counts on which the stability comparison rests are reported in Sect. 2.3.
Manuscript revision: Sect. 5.2 (lines 566–568) no longer claims greater sensitivity to initial conditions, random seeds, or run length. It now states only that NeuralGCM-evap and NeuralGCM-precip show stability challenges similar to those encountered in physics-based ESMs, which matter for multi-year integrations, and the completion counts are reported in Sect. 2.3.
Comment 3.28
Page 23, third paragraph. Please ensure that appropriate literature references are added to complete this paragraph.
Response: Agreed. We condensed this paragraph: the development-priority statements that lacked support were removed, and the remaining outlook cites Lee et al. (2024) and Ullrich et al. (2025) for PMP-based and DL-ESM evaluation and Bodnar et al. (2025), Cresswell-Clay et al. (2025), and Duan et al. (2025) for the planned extensions (Sect. 5.3, lines 570–582).
Minor revisions Comment 3.29
Page 2 line 28. It would be helpful to include a few concrete examples of standard weather-forecast scores, along with a brief note on their respective strengths and limitations.
Response: Agreed. We added concrete examples (Sect. 1, lines 55–63): RMSE summarizes error magnitude but mixes bias and variability and is scale-dependent; the anomaly correlation coefficient tests pattern or anomaly agreement but is insensitive to mean bias and can remain high with amplitude error; CRPS evaluates probabilistic calibration and sharpness but requires ensembles and can hide regional structure; the spread–skill ratio diagnoses ensemble dispersion but not spatial realism; and the Brier score evaluates binary-event probability accuracy but depends on threshold and event frequency.
Manuscript revision: These examples now motivate why forecast-oriented scores are necessary but insufficient for long-run Earth system evaluation, with citations to Brier (1950), Gneiting and Raftery (2007), and Rasp et al. (2024).
Comment 3.30
Page 3 line 24. Typo: please change to "Earth System Modeling".
Response: Agreed. We corrected the typo, now “Earth system modeling” following the lowercase convention for “Earth system” used throughout (line 110), and proofread the manuscript.
Comment 3.31
Page 9, line 11. A summary table for all the annual and seasonal climatology fields could be helpful. Additionally, increasing the font size in the Supplementary Figures would improve readability.
Response: We increased the font size and the readability of the panel labels in the supplementary figures.
Comment 3.32
Page 9, line 13. Could you provide an explicit formula or example showing how the global spatial RMSE is calculated?
Response: The global spatial RMSE is the area-weighted root mean square difference, RMSE = sqrt[sum_i w_i (M_i − O_i)^2 / sum_i w_i], evaluated on a common grid and valid-data mask, where w_i is proportional to the grid-cell area (cos φ_i on the regular grids used here) and M_i and O_i are the model and observed climatological values in grid cell i; it is computed separately for each season and for the annual mean.
Manuscript revision: The Fig. 2 caption states that errors are computed for each season over the globe for 1981–2013 after interpolating model and reference fields to a common 2.5° × 2.5° grid, and Sect. 4.1 describes the area-weighted climatologies.
Comment 3.33
Figure 2. Please consider increasing the font size for clarity. Rotating the matrix so that models are on the y-axis might also make the figure easier to read.
Response: We tried to rotate the matrix with a vertical orientation but the regenerated figure would take an entire page.
Comment 3.34
Page 10 line 10. In the sentence “We also notice the good performance of ACE2 across all the output variables”, the performance of ta-200 and ta-850 appears somewhat weaker; the authors may wish to nuance this sentence slightly.
Response: Agreed. We revised the sentence so that ACE2 is no longer described as performing well across all output variables; it now distinguishes the 200 and 850 hPa temperature errors from those of the remaining variables.
Manuscript revision: The sentence now reads: “ACE2 shows similar errors to NeuralGCM for 200 and 850 hPa temperature and slightly smaller errors for the remaining variables.” (Sect. 4.1, line 302).
Comment 3.35
Page 10 line 11. As "Root Mean Square Error" has already been introduced, the acronym "RMSE" can be used directly here.
Response: Agreed. We now use the previously defined acronym RMSE throughout (e.g., Sect. 4.1, line 289).
Comment 3.36
Figure 6. Could you please clarify the precise definition of "NAO skill"? It would also be helpful to discuss why the values simulated by both DL-ESMs over the North Pacific are notably higher.
Response: We replaced the ambiguous phrase “NAO skill” with the specific quantities shown: the spatial agreement of the NAO loading pattern, the fraction of variance explained by the common-basis-function mode, and the standard deviation of the associated principal component. The values in the panel titles are explained variances, not a universal NAO skill score, and the caption now states this. The enhanced North Pacific values may reflect leakage or mixing with Pacific circulation variability, forcing-related teleconnections, regional psl-reconstruction error, or a displacement of the modeled loading pattern; the current analysis cannot distinguish these mechanisms. The revised text discusses the feature as a regional pattern error and does not interpret it as an advantage of either model.
Manuscript revision: The Fig. 6 caption describes each row of panels and states that the percentage in each panel title is the explained variance, and the NAO paragraph of Sect. 4.2 (lines 361–372) defines the reported quantities and discusses the North Pacific loadings as a regional pattern error.
Comment 3.37
Figure 11. Adjusting the y-axis limits to zoom in on the active data range (as there are currently no data above 1 or below 10-3) would enhance visual clarity.
Response: We tried different arrangements to tighten the Fig. 11 y-axis but the current setting allows us to display both the legend and the period labels inside the panels.
Comment 3.38
Figure 13. Consider using insets or re-scaling the plot area to reduce unused white space and highlight the data.
Response: We tried to reduce unused white space and wanted to keep the same vertical scale for all the panels.
Citation: https://doi.org/10.5194/egusphere-2026-3295-AC3
Viewed
Since the preprint corresponding to this journal article was posted outside of Copernicus Publications, the preprint-related metrics are limited to HTML views.
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 162 | 0 | 3 | 165 | 0 | 0 |
- HTML: 162
- PDF: 0
- XML: 3
- Total: 165
- BibTeX: 0
- EndNote: 0
Viewed (geographical distribution)
Since the preprint corresponding to this journal article was posted outside of Copernicus Publications, the preprint-related metrics are limited to HTML views.
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
This study adopts the PCMDI Metrics Package to evaluate the forecasting performance of four deep learning-based prediction models for weather systems at various spatial scales, with intercomparisons against conventional CMIP models. The authors confirm that the AI-based models possess promising predictive skill yet exhibit distinct systematic error features, which they attribute to the limited time span of the training dataset. Although the findings can offer references for the evaluation research of artificial intelligence forecasting models, the analysis lacks sufficient depth. Most discussions merely describe intuitive features shown in the figures without linking them to the inherent disparities among different AI architectures. At minimum, the authors ought to discuss potential causes underlying performance gaps across multiple versions of the NeuralGCM model. Such insufficient and cursory analysis weakens the overall value of this work, and further in-depth elaboration on this part is strongly recommended.
Specific comments:
“These efforts offer important precedents, but they do not fully address the need for a comprehensive evaluation of DL-ESMs using the same diagnostics routinely applied to CMIP-class Earth System models.”
The authors are advised to clearly respond to their own research question: which extra evaluation metrics are supplemented to enable thorough evaluation of the DL-ESM model.
PCMDI: Do not assume that all readers are familiar with this abbreviation; its full name should be provided.
“We generated ensemble members by varying … For the stochastic NeuralGCM-evap and NeuralGCM-precip configurations, we additionally introduced perturbations through different random seeds.”
What are the underlying reasons for these different ensemble generation approaches? Please provide a detailed explanation of their respective purposes.
Table 2: What is the rationale for selecting the variables in Table 2? This selection serves as one of the key bases for justifying the claimed comprehensive evaluation.
“It also introduces a modest generalization test, because the boundary-condition source differs from the ERA5 training environment.”
While this generalization test is more suitable for physical models that strive for physical realism, AI models essentially deliver statistical predictions constrained by existing datasets. The bias correction schemes natively integrated inside the AI model may be incompatible with the boundary conditions set in this experiment. Hence, it remains debatable whether the evaluation metrics can truly characterize the real predictive and simulating performance of the AI model.
Figure 1: Both panels (c) and (d) are titled GPCP3.2. What are the specific differences between them? Do they correspond to different averaging periods?
Figure 4: Why are the results from the other two NeuralGCM model versions not presented?
Figure 6: A more thorough analysis is recommended to interpret the advantages of NeuralGCM, particularly whether the incorporated physical module governs the winter NAO pattern.