Evaluating Skill and Stability of ArchesWeather and ArchesWeatherGen under Multi-Decadal Climate Simulations

Singh, Renu; Brunstein, Robert; Jost, Antonia; Hasson, Yana; Rackow, Thomas; Monteleoni, Claire; Lessig, Christian; Couairon, Guillaume

doi:10.48550/arXiv.2605.29976

Preprints

https://doi.org/10.48550/arXiv.2605.29976

Preprints

10 Jul 2026

| 10 Jul 2026

Evaluating Skill and Stability of ArchesWeather and ArchesWeatherGen under Multi-Decadal Climate Simulations

Renu Singh, Robert Brunstein, Antonia Jost, Yana Hasson, Thomas Rackow, Claire Monteleoni, Christian Lessig, and Guillaume Couairon

Abstract. We evaluate the climate simulation capabilities of ArchesWeather and ArchesWeatherGen, two machine learning models originally trained for weather forecasting and evaluated up to a 10-day lead time. ArchesWeather is a deterministic model, while ArchesWeatherGen is a probabilistic flow-matching model leveraging ArchesWeather's forecasts, enabling ensemble based uncertainty quantification. In this work, we adapt these models to act as forced atmospheric models by using additional conditioning on the monthly mean sea surface temperature (SST) and sea ice cover (SIC) as boundary conditions. In particular, we follow the AI Model Intercomparison Project (AIMIP) Phase 1 protocol, which, analogous to the Atmospheric Model Intercomparison Project (AMIP), proposes a standardized experimental setup to evaluate the climate skill of machine learning based forced atmospheric models. We present a comprehensive evaluation of both models under these conditions, including comparison against numerical climate models, ablation studies that examine key design choices in the extension, and an analysis of forced versus unforced configurations. Despite being originally developed for weather forecasting, we demonstrate that forced configurations of ArchesWeather and ArchesWeatherGen produce stable long-term climate simulations, have a stable annual cycle, and capture the drift of many climate variables. The models faithfully reproduce ERA5's climatology, large-scale circulations and interannual variability, and they capture the tails of the distributions.

Received: 26 Jun 2026 – Discussion started: 10 Jul 2026

Renu Singh, Robert Brunstein, Antonia Jost, Yana Hasson, Thomas Rackow, Claire Monteleoni, Christian Lessig, and Guillaume Couairon

Status: final response (author comments only)

CEC1:
'Comment on egusphere-2026-3782 - No compliance with the policy of the journal', Juan Antonio Añel, 07 Aug 2026

Dear authors,
Unfortunately, after checking your manuscript, it has come to our attention that it does not comply with our "Code and Data Policy".
First, the "Code and Data Availability" section in your manuscript points to some GitHub sites. However, GitHub is not a suitable repository for scientific publication. GitHub itself instructs authors to use other long-term archival and publishing alternatives, such as Zenodo. Internally in the system, you have shared several repositories and sites containing additional information. The problem here is that the mentioned information is not available for Discussions, as it should be, and openly available.
The GMD review and publication process depends on reviewers and community commentators being able to access, during the discussion phase, the code and data on which a manuscript depends, and on ensuring the provenance of replicability of the published papers for years after their publication. Therefore, I am publishing here the information shared internally to the editors so that all the readers have access to it, and the Discussions stage is developed with guarantees:

Title: ERA5

DOI/URL: https://doi.org/10.24381/cds.adbb2d47

Title: AIMIP forcing dataset

DOI/URL: https://doi.org/10.5281/zenodo.16782372

Title: Geoarches

DOI/URL: https://doi.org/10.5281/zenodo.20784770

Access description: Dev branch in the github repo that will be merged shortly into the main branch. We will also be adding preprocessing code, model weights, and better rollout code to support reproduction. However, all the code to the train models is available (along with instructions for preprocessing).

Title: Geoarches Documentation

DOI/URL: https://geoarches.readthedocs.io/en/latest/archesweather/run/

Access description: General notebook to load and run the weather models, however a notebook specifically to run inference on the AIMIP versions will be added.

We can understand that you do not share the full ERA5 dataset used to train your models. However, if you have regridded the dataset, you must share here the software used to do it. Also, we can not accept readthedocs.io to store assets related to manuscripts submitted to Geosci. Model Dev. Therefore, the information in the readthedocs site must be included in a repository that we can accept.

Please, therefore, publish all the assets (code, data and documentation) used in your work in one of the appropriate repositories and reply to this comment with the relevant information (link and a permanent identifier for it (e.g. DOI)) as soon as possible. We cannot have manuscripts under discussion that do not comply with our policy.

The 'Code and Data Availability’ section of your preprint in arXiv must also be modified to cite the new repository locations, and corresponding references added to the bibliography.

I must note that if you do not fix this problem, we cannot accept your manuscript for publication in GMD.

Juan A. Añel

Geosci. Model Dev. Executive Editor

Citation: https://doi.org/10.5194/egusphere-2026-3782-CEC1
- CC1: 'Reply on CEC1', Robert Brunstein, 10 Aug 2026
  
  Dear Editor,
  
  thank your for pointing this out. We will address your concerns regarding data availability and repository structure and respond again immediately when we resolved this issue.
  
  Best regards,
  
  Robert Brunstein
  
  Citation: https://doi.org/10.5194/egusphere-2026-3782-CC1
CC2:
'Comment on egusphere-2026-3782', Robert Brunstein, 18 Aug 2026

Dear Editor,
Thank you for your comments. We address the concerns about suitable repositories below.
The code to download data, train and run ArchesWeatherGen for AIMIP is available in the following Zenodo repository: https://zenodo.org/records/20784771. The same Zenodo repository also includes a docs/ folder, which holds all the documentation from the readthedocs.io link on how to run and train models with the geoarches codebase, including an interactive notebook.

The ERA5 data was regridded while downloading from the Copernicus site, we have updated the Code and Data Availability statement with this information. We have also added information on how to access and regrid CMIP6 model outputs, as used in our for evaluation.
The required CMORisation and evaluation pipeline is available this Zenodo repository: https://zenodo.org/records/21913147. With that code, users are able to evaluate the output of our models against other climate models and ERA5.
We have updated the Code and Data Availability section in the arxiv preprint with references to both Zenodo repositories and information on how to access and re-grid data. The new arxiv version will be available from tomorrow on, we append our new Code and Data Availability section here:
“Code to train and run models is available at https://zenodo.org/records/20784771 (Couairon et al., 2026a). Model weights and configs will be made available on Hugging Face. ERA5 data for training is available for download on Copernicus (regridded to 1◦ resolution with the internal Copernicus tool) and scripts to preprocess are made available in the repo above. The data needed to rollout models (initial conditions and forcings) are available at https://zenodo.org/records/21952904. The reference climate model data from AMIP used in this study is available from https://wcrp-cmip.org/cmip-data-access/. The specific amip-p4k runs used for the warming scenarios can be found via https://doi.org/10.22033/ESGF/CMIP6.7536 (Danabasoglu, 2019) and https://doi.org/doi:10.22033/ESGF/CMIP6.8508(Silvers et al., 2018). We accessed all the data via the DKRZ compute cluster Levante. The models were regridded using the Climate Data Operators (CDO) (Schulzweida, 2023), https://doi.org/10.5281/zenodo.10020800. Forcing data is available at https://zenodo.org/records/17065758 (Arcomano et al., 2025). The model outputs are available in the AIMIP Phase 1 data, publicly available via the DKRZ S3 endpoint (Download instructions available in (Henn et al., 2026). Code to convert model outputs into CMOR format as required by AIMIP and to evaluate it successively is available at https://zenodo.org/records/21913147 (Brunstein and Jost, 2026)
Thank you for your time,
Robert and Renu, on behalf of all authors

Citation: https://doi.org/10.5194/egusphere-2026-3782-CC2
- CEC2:
  'Reply on CC2', Juan Antonio Añel, 18 Aug 2026
  
  Dear authors,
  Many thanks for your reply. I have checked it, and two issues remain outstanding regarding the compliance with the Code and Data Policy of the journal . First, we can not accept Hugging Face to store assets related to your manuscript. Also, we can not accept future expressions of compliance. The policy of the journal is clear regarding the fact that all the assets relevant for a submission must be published openly and without limitations at the submission time.
  Therefore, please, publish the mentioned information in a repository acceptable according to the policy of the journal, and reply to this comment with a new Code and Data Availability section that is in compliance.
  I must insist that if you do not fix this problem, we cannot continue with the peer-review process or accept your manuscript for publication in GMD.
  Juan A. Añel
  
  Geosci. Model Dev. Executive Editor
  
  Citation: https://doi.org/10.5194/egusphere-2026-3782-CEC2
  - CC3: 'Reply on CEC2', Robert Brunstein, 22 Aug 2026
    
    Dear Editor,
    
    we will address the remaining issues as fast as possible.
    
    Thanks for pointing them out again.
    
    Robert on behalf of the whole Team.
    
    Citation: https://doi.org/10.5194/egusphere-2026-3782-CC3
    
    AC1: 'Reply on CC3: Move model checkpoints to Zenodo', Renu Singh, 02 Sep 2026
    
    Dear Editor,
    To address the following concerns about data on Hugging Face and future expressions of compliance, we have added the model checkpoints to the Zenodo dataset link: https://zenodo.org/records/21952903.
    We have updated the Code and Data Availability section in the arxiv preprint. The main changes are (1) removing the line “Model weights and configs will be made available on Hugging Face” and (2) adding the fact that model checkpoints are also available at the Zenodo dataset resource.
    The updated arXiv version will be published soon, in the meantime we append our new Code and Data Availability section here:
    \codedataavailability{Code to train and run models is available at \url{https://zenodo.org/records/20784771} \citep{couairon2026geoarches}. ERA5 data for training is available for download on Copernicus (regridded to 1\unit{\degree} resolution with the internal Copernicus tool) and scripts to preprocess are made available in the repo above. The data needed to rollout models (model checkpoints, initial conditions, and forcings) are available at \url{https://zenodo.org/records/21952903}. The reference climate model data from AMIP used in this study is available from \url{https://wcrp-cmip.org/cmip-data-access/}. The specific amip-p4k runs used for the warming scenarios can be found via \url{https://doi.org/10.22033/ESGF/CMIP6.7536} \citep{https://doi.org/10.22033/ESGF/CMIP6.7536} and \url{https://doi.org/doi:10.22033/ESGF/CMIP6.8508} \citep{https://doi.org/10.22033/ESGF/CMIP6.8508}. We accessed all the data via the DKRZ compute cluster Levante. The models were regridded using the Climate Data Operators (CDO) \citep{schulzweida_2023_10020800}, \url{https://doi.org/10.5281/zenodo.10020800}. The model outputs are available in the AIMIP Phase 1 data, publicly available via the DKRZ S3 endpoint (Download instructions available in \citep{henn_2026_21302180}. Code to convert model outputs into CMOR format as required by AIMIP and to evaluate it successively is available at \url{https://zenodo.org/records/21913147} \citep{brunstein2026eval}. }
    Thanks, Robert and Renu
    
    Citation: https://doi.org/10.5194/egusphere-2026-3782-AC1
RC1: 'Comment on egusphere-2026-3782', Anonymous Referee #1, 16 Sep 2026

This paper demonstrates a relatively simple extension of the ArchesWeather prediction model to climate timescales by incorporating SST and sea ice forcings to its training protocol. The model shows good results in historical simulations, and impressive ability to produce stable and realistic out-of-sample +2K and +4K simulations. I would like to see some additional evaluation of physical constraints (especially air mass and water conservation) and some additional explanations of the protocol. Please see my attached full evaluation.

Citation: https://doi.org/10.5194/egusphere-2026-3782-RC1
RC2:
'Comment on egusphere-2026-3782', Anonymous Referee #2, 09 Oct 2026
Summary

This paper describes the extension of an existing combination of two ML weather models, the deterministic ArchesWeather with four seeds, and the flow-matching residual model ArchesWeatherGen, to the forced sea-surface temperature and sea-ice cover scenario using in the AIMIP Phase 1 protocol. The authors evaluate their combined model on 45-year rollouts against the 1-degree ERA5 ground truth. Their combined model performs impressively well on a variety of diagnostic criteria: the model remains stable over the evaluation period, generally captures the mean climatology well during the training period, captures small scale variability, and captures the interannual variability from the SST forcing in contrast to the biased unforced model.

Comments:

I overall found the paper to be well-written, organized and readable. However, there are a few strong claims in the paper that I found were not so well supported, or were statistically weak, that I discuss below. If these are revised, then I would recommend this paper for publication.

The abstract claims that the models capture the drift of many climate variables and faithfully reproduce interannual variability and the tails of distributions. Given the results shown in the paper, these statements should be tempered.

The temperature trend is captured fairly well by both models in the training period, but Sec. 4.1 then states that the models “fail to fully capture this increase in the trend” during the test period. Moreover, because the trend largely depends on the SST, which is prescribed, it would be much more convincing to show the trend over land only, or even a global map of the trend over each land grid point.

The interannual variability claim is based on the evaluation of the SOI, where both models appear to overestimate the power in the 2-7 year range and underestimate it at higher frequencies. It’s possible that this is explained by the fact that, as stated in Sec. 4.4, the SOI is computed from an ensemble mean rather than individual ensemble members. An updated figure should also show the mean and spread computed over each member.

The claims on extremes performance are based on two types of extremes, wind speed and surface temperature, on the global scale. But the claim in the abstract says nothing on whether extremes are captured regionally at the correct frequency, or how well extremes in other variables are captured.

In the annular mode analysis, the authors select the ensemble member with the highest pattern correlation. I would prefer to see comparisons across all ensemble members, with confidence intervals shown, to support the conclusion that the ML models are on par with or superior to the numerical ones in their replication of the annular modes.

Similar to my comment on the trend in ERA5, it would be very interesting to see the change in the land temperature trend in the +2K and +4K experiments (as done in the AIMIP Phase 1 plots). If the strongest warming largely occurs over the southern ocean, and not over the land as seen in physics-based model simulations, this should be shown and discussed.

A more minor comments on the results of the paper:
The claims of the monsoon test rely on only 1 year; it would be much more convincing to show these results over the entire test period.

Fig. 6 is missing units on the spatial frequency axis.

The authors should reference prior work that discusses the Cramér–von Mises statistic and its relationship to CRPS (Gneiting and Raftery 2007; Rizzo and Székely 2016). I find the author’s statement that they “contribute” this way to compute this distance score to overstate its novelty, but this is mostly an issue of wording.

Some details of the AIMIP Phase 1 protocol that are relevant to this paper are missing here. These include how exactly the monthly SST and SIC are interpolated and from which dates, and how exactly the data was regridded and with what tool.

Proofreading corrections (other reviewers may have found these already):
Author list: "Couiaron" should read Couairon

Sect. 1 says "40 year rollouts"; Sect. 3.2 and Sect. 5 say 45 years; Oct 1978 to Dec 2024 is about 46 years.

Sect. 3.1: "easwart" (eastward)

Sect. 4.1: "ArchesWeatherGen (−0.05 K and 4.17 kg kg−1)" is missing the exponent (4.17 × 10⁻⁵).

Eq. 1: the product index and the conditioning set (F₁…F_N vs F_{2δ:Nδ}) are inconsistent.

Typos: "northen", "ERA55", "ut with", "ration" (ratio), "evalation", "Certain outputs is", "after a after", "ArchesWeatherGenu".

Missing "Fig." in cross-references: "close to ERA5 7", "In 12 a)", "in 11".

Fig. 5: title "Surface Air Temperature (850 hPa)" should be "Air temperature (850 hPa)".

Fig. 11 discussion refers to a "ridge-like structure over the pacific ocean", but the maps only cover 50–100°E. Should this be the Indian Ocean?

Some acronyms are not defined (NWP, OOD, DOY, CMOR, ViT, PSD).

"2 K or 4 K degrees warmer": drop "degrees".
Citation: https://doi.org/10.5194/egusphere-2026-3782-RC2

Renu Singh, Robert Brunstein, Antonia Jost, Yana Hasson, Thomas Rackow, Claire Monteleoni, Christian Lessig, and Guillaume Couairon

Data sets

ERA5 Bill Bell, Hans Hersbach, Adrian Simmons, Paul Berrisford, Per Dahlgren, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Raluca Radu, Dinand Schepers, Cornel Soci, Sebastien Villaume, Jean-Raymond Bidlot, Leo Haimberger, Jack Woollen, Carlo Buontempo, and Jean-Noël Thépaut https://doi.org/10.24381/cds.adbb2d47

AIMIP forcing dataset Troy Arcomano, Brian Henn, and Christopher Bretherton https://doi.org/10.5281/zenodo.16782372

Model code and software

Geoarches Guillaume Couairon, Renu Singh, and Robert Brunstein https://doi.org/10.5281/zenodo.20784770

Interactive computing environment

Geoarches Documentation Renu Singh, Aymeric Delefosse, and Adrien Le Coz https://geoarches.readthedocs.io/en/latest/archesweather/run/

Renu Singh, Robert Brunstein, Antonia Jost, Yana Hasson, Thomas Rackow, Claire Monteleoni, Christian Lessig, and Guillaume Couairon

Viewed

Since the preprint corresponding to this journal article was posted outside of Copernicus Publications, the preprint-related metrics are limited to HTML views.

Total article views: 357 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	BibTeX	EndNote
353	0	4	357	0	0

HTML: 353
PDF: 0
XML: 4
Total: 357
BibTeX: 0
EndNote: 0

Views and downloads (calculated since 10 Jul 2026)

Month	HTML	PDF	XML
Jul 2026	141	1	142
Aug 2026	71	0	71
Sep 2026	80	2	82
Oct 2026	61	1	62

Cumulative views and downloads (calculated since 10 Jul 2026)

Month	HTML	PDF	XML
Jul 2026	141	1	142
Aug 2026	71	0	71
Sep 2026	80	2	82
Oct 2026	61	1	62

Viewed (geographical distribution)

Since the preprint corresponding to this journal article was posted outside of Copernicus Publications, the preprint-related metrics are limited to HTML views.

Total article views: 248 (including HTML, PDF, and XML) Thereof 248 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 21 Oct 2026

Short summary

We investigate if two machine learning weather models, ArchesWeather and ArchesWeatherGen can simulate long-term climate trends. We adapt the models to be driven by observed sea surface temperature (SST), and perform predictions 1978–2024. We show they capture important aspects of observed climate and respond significantly to artificially warmed SST, comparable with reanalysis and physics-based climate models. They require significantly less compute resources than traditional climate models.


Total:	0
HTML:	0
PDF:	0
XML:	0