the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 11 Jul 2025
Rapid Evaluation Framework for the CMIP7 Assessment Fast Track
Abstract. As Earth system models (ESMs) grow in complexity and in volumes of output data, there is an increasing need for rapid, comprehensive evaluation of their scientific performance. The upcoming Assessment Fast Track for the Seventh Phase of the Coupled Model Intercomparison Project (CMIP7) will require expeditious response for model analyses designed to inform and drive integrated Earth system assessments. To meet this challenge, the Rapid Evaluation Framework (REF), a community-driven platform for benchmarking and performance assessment of ESMs, was designed and developed. The initial implementation of the REF, constructed to meet the near-term needs of the CMIP7 Assessment Fast Track, builds upon community evaluation and benchmarking tools. The REF runs within a containerized workflow for portability and reproducibility and is aimed at generating and organizing diagnostics covering a variety of model variables. The REF leverages best-available observational datasets to provide assessments of model fidelity across a collection of diagnostics. All diagnostics were identified and finally selected with community involvement and consultation. Operational integration with the Earth System Grid Federation (ESGF) will permit automated execution of the REF for specific diagnostics as soon as model data are published on ESGF by the originating modelling centres. The REF is designed to be portable across a range of current computational platforms to facilitate use by modelling centres for assessing the evolution of model versions or gauging the relative performance of CMIP simulations before being published on ESGF. When integrated into production simulation workflows, results from the REF provide immediate quantitative feedback that allows model developers and scientists to quickly identify model biases and performance issues. After the REF is released to the community, its subsequent development and support will be prioritized by an international consortium of scientists and engineers, enabling a broader impact across Earth science disciplines. For instance, the REF will facilitate improvements to models and reductions in uncertainties for projections since ESMs are the main tool for studying the global Earth system. Production of reproducible diagnostics and community-based assessments are the key features of the REF that help to inform mitigation and adaptation policies.
Competing interests: At least one of the (co-)authors is a member of the editorial board of Geoscientific Model Development.
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this paper. While Copernicus Publications makes every effort to include appropriate place names, the final responsibility lies with the authors. Views expressed in the text are those of the authors and do not necessarily reflect the views of the publisher.- Preprint
(944 KB) - Metadata XML
- BibTeX
- EndNote
Status: open (until 23 Oct 2025)
- RC1: 'Comment on egusphere-2025-2685', Anonymous Referee #1, 05 Sep 2025 reply
Data sets
Observational Data for use by the REF Earth System Grid Federation (ESGF) https://esgf.github.io/nodes.html
Model code and software
Climate Rapid Evaluation Framework (REF) REF Delivery Team https://github.com/Climate-REF/climate-ref
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 2,036 | 125 | 16 | 2,177 | 28 | 29 |
- HTML: 2,036
- PDF: 125
- XML: 16
- Total: 2,177
- BibTeX: 28
- EndNote: 29
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Comments
Section 4.3: I’m not exactly sure of the difference between the ‘variables’ described here and the ‘diagnostics’ referred to in Table 1. For example, all the diagnostics in the Land and Land Ice Realm are objects I would think of as variables output by a land model (GPP, LAI etc). I do see how some of the diagnostics (like ECS/TCR etc in the ES realm) would be computed from variables and not be variables themselves. Perhaps it would be good to clarify that there might or might not be duplication/cross-over between ‘diagnostics’ and ‘variables’.
Section 5.3: this is more of a philosophical thought about subsetting models for particular scientific purposes and/or regional downscaling – for an end user, it would be tempting to just grab the models that perform well on certain regions/realms of interest, but if those same models are way off in other regions/realms, would that be a fair way to subset models? Perhaps this is more of a question for the community as a whole to come up with the best way of pulling a subset of models to look at specific questions (e.g. have some kind of penalty for not doing well for other regions/variables).
Table B1: (i) I am curious why cSoil was included but not cVeg? Is there a plan to include more datasets such as those from GEDI? (ii) would it be possible to add the dates over which the reference datasets are available? I am familiar with the ILAMB land datasets and I think they only go up to 2014 – have there been extensions to these for more recent years?
Section C1 1.1: what is the frequency of this calculation? i.e. annual/decadal/something else?
Minor Edits
Line 114: “produced, processes” -> “produced, and processes”
Line 117: what is a “realm” referring to here? These are defined in line 295 but should be introduced earlier here.
Figure 1 caption: spell out the acronyms (e.g. I am not sure what ‘DAG’ is)
Line 161: “a” -> “an”
Line 175: “at” -> “by the”
Line 259 -> add comma between “(ENSO)” and “CLIVAR”
Line 734: ref should be in parentheses
Line 743: insert “by” after “influenced”
Line 773: insert “by” after “obtained”
Line 780: I am not sure what is meant by “brought direction”
Line 792: “patterns” -> “pattern”
Line 802: insert “in” after “patterns”