the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 14 Jan 2025
The Coupled Model Intercomparison Project (CMIP): Reviewing project history, evolution, infrastructure and implementation
Abstract. The CMIP6 project was the most expansive and ambitious Model Intercomparison Project (MIP), the latest in a long history, extending back four decades. CMIP has captivated and engaged a broad, growing community focused on improving our climate understanding. It has anchored our ability to quantify and attribute the drivers and responses of the observed climate changes we are experiencing today.
The project's profound impact has been achieved by combining the latest climate science and technology. This has enabled the production of latest-generation climate simulations and the dissemination of their output, which has seen increased community attention in every successive phase. The review emphasizes the pragmatics of progressively scaling up efforts, the evolution of how the MIPs were implemented, and the coordinated efforts to establish a minimal infrastructure to make that possible, most recently delivering CMIP6.
Competing interests: At least one of the (co-)authors is a member of the editorial board of Geoscientific Model Development.
Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.- Preprint
(1390 KB) - Metadata XML
- BibTeX
- EndNote
Status: final response (author comments only)
-
CC1: 'Comment on egusphere-2024-3729', Bjorn Stevens, 20 Jan 2025
Publisher’s note: the content of this comment was removed on 20 January 2025 since the comment was posted by mistake.
Citation: https://doi.org/10.5194/egusphere-2024-3729-CC1 -
CC2: 'Comment on egusphere-2024-3729', Bjorn Stevens, 20 Jan 2025
The paper provides a wealth of information about the mechanics of CMIP and in this sense is a very useful documentation of these activities and the present status by many of the key figures involved. The summary of the provision and usage of CMIP data and development of the infrasturcute was particularly useful and novel. While, as the paper’s introduction states, many aspects of CMIPs history have been shared before, by many of the same authors, this telling is more comprehensive and serves its stated purpose of providing a uniform treatment. From a more critical perspective the paper wades into value judgements that it fails to substantiate, when these are combined with its overall uncritical outlook, it comes across as needlessly self congratulatory, which detracts from its more meritorious contributions.
A defining quality of scientific work, e.g., as described by, the Mertonian norms, is organized skepticism. The present contribution would be strengthened if this were in greater evidence. For instance the articles by myself and Jakob et al. are dismissed in passing, at the end, without addressing the question they raise, which is whether the continued growth of CMIP as a quasi operational activity (Meehl’s words) that uses a research, rather than an operational, infrastructure is good for research or helpful for society. By binding research to a quasi operational activity what burden does CMIP place on free an innovative research? How effective is it in addressing feedback from downstream users (consider that climate services in Africa are still based on downscaling of CMIP5 data), e.g., Jakob et al? Why isn’t it leading to markedly better models? Can the growing demands on the infrastructure continue to be borne by the research community or does it need a more permanent footing, as was, for instance advocated for by EVE.
Relatedly, the manuscript makes many unqualified and unsupported statements about the impact and success of CMIP. Consider the statement (l1082): "CMIP has generated profound scientific insights that define how we understand and address climate change and our ability to quantify and attribute the drivers and responses to the observed climate changes we are experiencing today." Related to this is the statement in the abstract that "CMIP has captivated and engaged a broad, growing international community focused on improv-ing our climate understanding" or reference to the CMIP project’s central goal "to advance scientific understanding of the Earth system and its responses to ongoing natural and anthropogenic forcing agents" (line 349) or most grandiosely "However, without CMIP, the IPCC assessments could not have been possible. Without the coordinated community climate science efforts embodied by the AMIP and CMIP phases, progress in Earth’s climate understanding would not have advanced to our present state of knowledge."
The manuscript clearly demonstrates that there is a lot of CMIP data and that this data is widely used. This is certainly one measure of impact. I also think the progression in understanding through the first three CMIP phases is reasonably well documented. But with much of the growth occurring after CMIP3 the critical question is what — scientifically — has been wrought of the additional effort. There is a common perception, which I share, that what was to be learned from CMIP was mostly learned in CMIP3, and CMIP5’s contribution was to confirm that. Given the growth between CMIP5 and CMIP6, and its associated cost, what is the incremental gain? Certainly updating the scenarios is a necessary and valuable activity, but is it well served by the uncertain and adhoc time-lines, and does it need to be done through an activities like CMIP. The manuscript doesn't answer these questions. It doesn't have to answer these question, unless it wants to continue to attach value to the mechanics it describes. And in that case it must do so in a systematic and critical way. This requires addressing some of the arguments in my essay, for instance why some of the biggest and most lauded steps forward happened in areas where model output was disregarded, as for the case of climate sensitivity, and why there is little evidence of model improvement, beyond the null hypothesis that nominal gains are from improved resolution and gaming the diagnostics.
Even if the paper limits itself to a discussion of the mechanics of CMIP,. rather than a valuation of these mechanics, a skeptical point of view would be refreshing. What hasn't worked? What should be done differently even if the authors are of the mind that CMIP should be an activity that remains a service of the research community, i.e., is pursued using a research infrastructure. Has the considerable investments in automated evaluation tools, reprocessing of satellite data to align with model output, and model documentation activities really been beneficial? If so this must be demonstrate rather than just claimed.
One disadvantage of large author lists where most of the author’s don’t meaningfully contribute, is that it makes it difficult to develop a critical outlook, and this is certainly the case here. It also undermines good scientific practice. Surely the standard for authorship of a scientific paper requires more than having "contributed to the final version of the manuscript." Considering how fewer than half the authors are recognized for specific contributions, even contributions as small as commenting on a section, it makes the contributions of the non listed authors seem even more trivial. Given that many people who are not authors contributed to the design and execution of various phases of CMIP, not to mention the many modelling groups, a higher bar on authorship seems warranted.
Rather more editorial comments:
As a minor and historical note, when discussing the DECK, I suggested it as a component of Meehl's (as best I can recall) caricature of a CMIP6 prototype. If my memory serves me well Meehl proposed a prototype version of CMIP6 as a sailing ship of discovery, with different sails representing different MIPS. Based on this I proposed the DECK as the place that secured the masts for the sails, and as such as a specific response to an earlier (but unreferenced) criticism of CMIP by Marotzke and Rauser. The critique being that CMIP phases lacked continuity. This was an example of how CMIP incorporated and adapted to criticism.
In many places this reader gets the feel that quantity is a measure of quality. This works for describing water, but not for scientific advancements
The authors may want to guard against the impression, which is evident at the end, even though it is dealt with much better earlier in the manuscript, that somehow model intercomparison is an activity that started with AMIP.
I am not sure what the authors have in mind when they refer to CMIP6 models having fewer parameterizations (l570). Maybe this could be made explicit.
-- Bjorn Stevens
Citation: https://doi.org/10.5194/egusphere-2024-3729-CC2 -
CC3: 'Comment on egusphere-2024-3729', Rasmus Benestad, 06 Feb 2025
Thank you for this account of the CMIP and its history. It’s very nice to get such a detailed description.
The authors could perhaps acknowledge the KNMI Climate Explorer (https://climexp.knmi.nl/start.cgi) that also compiled CMIP data and provided them in a standard structure that more easily enabled the analysis of large multi-model ensembles. It reduced the technical barriers to using large numbers of global climate model (GCM) runs.
And perhaps this discussion could include some reference to analysis of large multi-model ensembles as a collective entity, as opposed to comparing individual models (DOI:10.1016/j.cliser.2017.06.013). I’m not sure if people see this subtle difference, and most evaluations involve comparing individual model runs as opposed to how the combined statistics of multi-model ensembles compare with the real world (e.g. standard statistical tests of distributions against observations). There have been some examples where entire CMIP3 (DOI:10.1175/2010JCLI3687.1), CMIP5 (DOI:10.1088/1748-9326/11/5/054017 & DOI: 10.5194/essd-9-905-2017) and CMIP6 (DOI:10.5194/hess-29-45-2025) multi-model ensembles have been downscaled through empirical-statistical downscaling (ESD) which also provide a means for evaluating them through the calibration and diagnostics of the statistical models. Such large ensembles may furthermore capture stochastic regional decadal variability (Deser et al., 2012; 2020; DOI:10.1038/nclimate1562/10.1038/s41558-020-0731-2), and are needed for robust assessments for both impact studies and climate change adaptation.
There are recent evaluations of CMIP GCMs beyond those cited by the authors, e.g. in terms of their ability to reproduce the spatio-temporal covariance structure (DOI:10.5194/gmd-16-2899-2023) and important aspects of the global hydrological cycle (10.1038/s41612-024-00794-z). These contributions may perhaps be relevant for the discussion here. Also the R-shiny app ‘GCMeval’ (DOI:10.1016/j.cliser.2020.100167) was developed for an interactive evaluation of CMIP GCMs, and may merit a mention.
The academic literature citations often miss out important studies which go under the radar and exaggerate the importance of papers written by ‘popular’ scholars. There is a tendency that friends cite each other more and miss out relevant contributions from the wider community, which may be a natural consequence because they talk to each other and learn about each other's work. It’s important to refer to relevant work, even from a wider circle, to ensure highest quality and to acknowledge the work done by colleagues regardless whether they belong to the inner circle or the wider community. It’s hard to keep abreast with all publications, and it may even be a problem with such a large output in terms of publications, but perhaps AI and search tools may help in identifying those that ought to be considered in the literary review/discussion.
Citation: https://doi.org/10.5194/egusphere-2024-3729-CC3 -
CC4: 'Comment on egusphere-2024-3729', Cath Senior, 07 Feb 2025
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2025/egusphere-2024-3729/egusphere-2024-3729-CC4-supplement.pdf
-
RC1: 'Comment on egusphere-2024-3729', Cath Senior, 10 Feb 2025
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2025/egusphere-2024-3729/egusphere-2024-3729-RC1-supplement.pdf
-
EC1: 'Comment on egusphere-2024-3729', Juan Antonio Añel, 12 Feb 2025
Dear authors,
Please, for new versions of your manuscript remember modifying the "Code and Data Availability" section of your manuscript and providing a repository (link and DOI) for the scripts you mention there. Currently, it reads "Data underpinning figures in the paper, in addition to tabulated additional information can be viewed in a1110 paper-dedicated GitHub repository at https://github.com/durack1/CMIPSummary, or online using NBViewer at https://nbviewer.org/github/ durack1/CMIPSummary/blob/main/figuresAndTables.ipynb."
GitHub is not a suitable repository for scientific publication. GitHub itself instructs authors to use other long-term archival and publishing alternatives, such as Zenodo. Please, check our "Code and Data Policy" (https://www.geoscientific-model-development.net/policies/code_and_data_policy.html) for additional details.
Also, please, reply in advance to this comment with the details for the repository requested.
Juan A. Añel
Geosci. Model Dev. Executive Editor
Citation: https://doi.org/10.5194/egusphere-2024-3729-EC1 -
RC2: 'Comment on egusphere-2024-3729', Anonymous Referee #2, 20 Feb 2025
In this manuscript, the authors present the development of the support infrastructure from the beginning of the CMIP project to its last phase, CMIP6, to accompany the evolution of this project, the increase in the number of simulations, the number of models, the number of users, etc. It is built around the history of this project, which is then broken down into different aspects. It clearly shows the difficulty and scale of the task involved in producing these simulations and making their results available to an ever-growing community of users. This is a well-constructed, well-written article that will be a useful addition to the existing literature on CMIP.
The authors rightly emphasise the contribution of CMIP, which promotes and facilitates the possibility of carrying out identical experiments with several models, following the same protocol and giving open access to the results of these simulations for critical and open analysis. This has enabled a large community of scientists, not directly involved in climate modelling, to analyse these simulations and carry out numerous studies, for different purposes: climate variability, detection-attribution, climate sensitivity, etc.
How to assess the impact and success of such a project is difficult and may therefore be open to criticism. Nevertheless, the criteria used by the authors are those that are commonly used, and the results do not contradict my perception of the impact of CMIP. For example, I find it interesting that the articles presenting each of the MIPs are generally cited more than 100 times, which indicates, for each of these projects, the support of a significant number of scientists.
The fact that the number of models and the number of countries involved in CMIP6 has almost doubled compared with CMIP5 (and tripled compared with CMIP3) (see Figure 1) seems to me to be an indicator that climate modelling and climate change simulations are being taken on board by an increasingly large community, even though probably few (if any) models are new. One suggestion would be to add, for example, a table listing these countries to have a better image of this spreading.
In conclusion, I recommend publishing this manuscript in GMD after some minor changes.
More specific comments
The authors mention that the PMIP has emerged in parallel with the phases of the A/CMIP since 1991, but rather late in the text, without including this project in the dynamics of the emergence of the CMIP (section 2.2), although I think they should do so.
The authors mention several times the pressure that the CMIP project puts on the teams that develop climate models. When resources are limited, there is not necessarily much time to develop and improve models between two phases of CMIP, especially with current models which are increasingly complex, incorporating more and more processes. This pressure can slow down the improvement of models and/or prevent risk-taking in their development. This is not a new issue, but it is nevertheless worth mentioning, especially at a time when the modelling landscape is changing rapidly.
Details:
lines 436-444: something is wrong here, part of the text is duplicated
Figure 5: Apart from the first 3 colours and the last one, it is difficult to identify which colour corresponds to which set of experiences. It would be useful to improve legibility, for example by adding hatching for some of the colours.
Figure A1: is it the number transferred files, like in Figure 5?
Citation: https://doi.org/10.5194/egusphere-2024-3729-RC2 -
CC5: 'Comment on egusphere-2024-3729', Annalisa Cherchi, 11 Mar 2025
I think it is overall nice to have the history and evolution of CMIP but I have a couple of general comments on the structure of the manuscript.
In primis, I think that given the past it would be nice to have more information and evaluation of what has been changed/is changing in CMIP7 compared to CMIP6 (in terms of infrastructure and implementation). At the same time, even if it could be a wide discussion it would be nice to see more about what has been the gain of the massive effort dedicated to CMIP6 compared to previous rounds (not only in terms of amount of data and/or papers published). Something could be eventually summarised/reviewed from the IPCC AR6 WG1 report itself. In addition, I think that some discussion/considerations should be dedicated to the data handling: are we expecting even more data out of CMIP7 compared to CMIP6? if yes, is this huge amount of data sustainable?
Second, my understanding is that infrastructure here is mostly associated with the ESGF node, but my understanding following on CMIP webinar/newsletter/ideas is that in CMIP7 infrastructure could be much more than the ESGF node (evaluation and operationalization(?)). Is this understanding correct?
Other minor comments:
- in most places acronyms are explicit more than once, I think first time for each acronym could be enough;
- line 34: actually for some variables and processes we don't have much self-confidence on observations as there could be uncertainties even in what is observed (this is mostly a detail not sure how/if could be inserted here)
- line 288-291: still the timeline (between CMIP6 experiments available and IPCC deadlines) was not well aligned (there have been delays in the assessment of the projections for example as not much literature with CMIP6 models outputs was ready yet)
- lines 660-666: infrastructure mostly identified as ESGF node (see also general comment above)
- lines 799-800: reference is missing
- Section 4.6: references as outcomes of IS-ENES are missing here
- overall a general check of the manuscript would be needed, there are repetitions of concept in some places that could be reduced, if not avoided
Citation: https://doi.org/10.5194/egusphere-2024-3729-CC5 -
RC3: 'Comment on egusphere-2024-3729', Ben Santer, 12 Mar 2025
This is a comprehensive summary of the history of CMIP - the Coupled Model Intercomparison Project. I'm not aware of a similarly comprehensive CMIP summary that is available in the published literature. I strongly recommend publication of the paper subject to revision.
My principal comments relate to the "balance" of the paper. The longest section (Section 4) deals with "CMIP supporting infrastructure and organizations". While this section provides interesting and important technical information on the "nuts and bolts" of CMIP5, I would have preferred to see a shorter Section 4 and an expanded Section 6 (CMIP impact). In my opinion, an expanded section on the scientific impact of CMIP will be of greater interest to readers.
At present, Section 6 focuses primarily on metrics of the "uptake" of CMIP information by the scientific community (e.g., data downloads and citations of publications relying on CMIP simulation output). There is limited discussion of the scientific "lessons learned". For example, what did CMIP - and other MIPs - teach us about the causes of model systematic errors? What did it teach us about changes in model skill over all generations of AMIP and CMIP? What were some of the major scientific success stories? Did CMIP systematically improve the fidelity with which the mean state, seasonal cycle, and properties of key modes of internal variability are simulated? Did CMIP help to narrow uncertainties in model-derived estimates of climate sensitivity? What are some of the scientific issues that CMIP has not helped to resolve?
I've been a practitioner of climate change detection and attribution ("D&A") over the full 30+ year sweep of the history of CMIP. My personal perspective is that CMIP has been highly influential in D&A research, and has provided critical support for the IPCC AR6 finding of "unequivocal" human influence on the atmosphere, oceans, and land surface.
CMIP allowed D&A analysts to transition from a single-model perspective to a multi-model perspective. It enabled us to assess whether the identification of human fingerprints on climate is internally and physically consistent across a range of different variables, and whether a "discernible human influence" on global climate is robust to uncertainties in model estimates of external forcing, the responses to external forcing, and the properties of intrinsic variability.
More recently, the large initial condition ensembles submitted to CMIP5 and CMIP6 (which receive little mention in the paper) have greatly facilitated estimation of forced signals in model simulation output, and the regression-based removal of such forced signals from observations. This in turn has made it easier to compare observed (but uncertain) intrinsic variability with model results, and to explore the important question of whether external forcing modulates internal variability.
CMIP has also improved scientific understanding of the uncertainties in key historical anthropogenic and natural external forcings. These forcings have evolved markedly from CMIP5 to CMIP6, and will continue to evolve in CMIP7. This evolution in scientific understanding allows D&A analysts to assess whether anthropogenic fingerprint identification in observations is robust to forcing uncertainties.
Additionally, results from CMIP have revealed significant problems with observational datasets, such as satellite-based estimates of tropospheric temperature change (see, e.g., Santer et al., 2021). One "lesson learned" from CMIP is that confronting models with data can improve both ESMs and observations - not just the models alone.
I would contend that another success story from CMIP is the ability to better quantify the contributions of different individual forcings to simulated and observed climate changes. I'm thinking in particular of Figures SPM.1b and SPM.2 in the IPCC AR6 Working Group I Report. And CMIP allows us to consider whether there are any robust, physically-interpretable emergent constraints on uncertainties in projections of future climate change (and to place error bars on such projections).
I'm not recommending that the authors explore all of these issues in Section 6. I do, however, think that they should highlight a few key scientific success stories in Section 6, above and beyond providing download metrics and citation numbers. Such specific success stories are necessary to support the case that the authors are making - that the scientific value of CMIP outweighs the "opportunity cost" of participation.
I am appending an edited version of the manuscript. I've made a number of suggestions to improve the clarity of the text. I hope these suggestions are helpful.
Signed review: Ben Santer.
Santer, B.D., S. Po-Chedley, C. Mears, J. Fyfe, N. Gillett, Q. Fu, J. Painter, S. Solomon, A.K. Steiner, F.J. Wentz, M.D. Zelinka, and C.-Z. Zou, 2021: Using climate model simulations to constrain observations. Journal of Climate, 34, 6281-6301. https://doi.org /10.1175/JCLI-D-20-0768.1
Viewed
| HTML | XML | Total | BibTeX | EndNote | |
|---|---|---|---|---|---|
| 832 | 162 | 15 | 1,009 | 10 | 9 |
- HTML: 832
- PDF: 162
- XML: 15
- Total: 1,009
- BibTeX: 10
- EndNote: 9
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 409 | 40 |
| France | 2 | 84 | 8 |
| United Kingdom | 3 | 78 | 7 |
| Germany | 4 | 71 | 7 |
| China | 5 | 37 | 3 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 409