the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 24 Nov 2023
Objective Evaluation of Earth System Models: PCMDI Metrics Package (PMP) version 3
Abstract. Systematic, routine, and comprehensive evaluation of Earth System Models (ESMs) facilitates benchmarking improvement across model generations and identifying the strengths and weaknesses of different model configurations. By gauging the consistency between models and observations, this endeavor is becoming increasingly necessary to objectively synthesize thousands of simulations contributed to the Coupled Model Intercomparison Project (CMIP) to date. The PCMDI Metrics Package (PMP) is an open-source Python software package that provides "quick-look" objective comparisons of ESMs with one another and with observations. The comparisons include metrics of large- to global-scale climatologies, tropical inter-annual and intra-seasonal variability modes such as El Niño-Southern Oscillation (ENSO) and Madden-Julian Oscillation (MJO), extratropical modes of variability, regional monsoons, cloud radiative feedbacks, and high-frequency characteristics of simulated precipitation, including extremes. The PMP results are produced in the context of all model simulations contributed to CMIP6 and earlier CMIP phases. An important priority of the PMP is to document evaluation statistics for all Historical and AMIP simulations submitted to recent phases of CMIP, providing version-controlled information for all data sets and software packages being used. Among other purposes, this also enables modeling groups to assess performance changes during the ESM development cycle in the context of the error distribution of the multi-model ensemble. In this paper, we present an overview of the PMP including its history to date, capabilities, recent updates, and future direction.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(3528 KB)
-
Supplement
(1004 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(3528 KB) - Metadata XML
-
Supplement
(1004 KB) - BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2720', Anonymous Referee #1, 04 Dec 2023
In their manuscript, Lee et al. present the PCDMI Metrics Package (PMP) version 3, which is an open-source Python software package that provides tools for comparisons of ESMs with each other as well as for comparisons of ESMs with observations. A large range of atmospheric processes can be assessed with this software package. Since this package has been developed within the CMIP comparison projects the PMP results can be produced for all model simulations contributing to CMIP6 and earlier CMIP phases.
The manuscript is generally well written and the package useful for the scientific community, but I have several comments that should be taken into account before publication in GMD. Generally, the authors should take care that their message comes better through. At the moment, I have the feeling that there is a lot written, but what is the take home message? Is this a unique package or are there other, similar packages available. What is new or unique for your specific software package?
General comments:
- This is a quite extensive overview and I was wondering how a new user should get started with PMP. I saw from the links you provided that everything is quite well documented, however, this is so much information at once that I think that a new user will have really trouble getting started. Is there e.g. a documentation that could be downloaded as a single pdf file or does one really spend hours reading everything on the computer screen?
- How can a user install PMP on a computer? What requirements are needed? You provide a link to your github repository, where this information is provided, but I think this information should also be provided in the manuscript itself.
- A general questions I was wondering about is, if you are offering workshops for training of new users?
- In the abstract you state what you will discuss the history up to date, recent updates and future directions. The future directions are discussed to some part in Section 6, but I could not find any information or discussion on the history of PMP up to date or what the updates between former and this version were.
- Generally, all sections seem to be a bit too lengthy and to my opinion not really coming to the point. Best example is the summary and future directions section. Although you have a discussion section, you provide a quite long summary and future directions section without really summarizing what you have presented.
- Is this the only tool for analyzing CMIP data or have there also other tools been developed?
Specific comments:
P2, L47-48: When was version 1 developed (published)? What are the major changes/new developments you are presenting here?
P3, L82: Also here you should clearly state when the first version was developed. Has this version been somewhere documented/published? Or has this version just been provided to the CMIP community?
P4, L86: Here, some examples should be given. Which performance metrics or statistical measures are used?
P4, L101: Also here, add some examples.
P4, L102: Why only a subset? Specify.
P4, L102: What do you mean with CMIP class models? Models that participate in CMIP or the kind of models participating in CMIP?
P4, L109: References? Are there any publications?
P6, L170: The following subsections are rather the “processes” that can be assessed with PMP than metrics. The statistics you are using are the metrics for each of this processes.
P6, L174: “well-established statistics” -> You should more clearly write here which statistics are used (at least some examples should be given).
P6, L178: Provide here a typical example. What is a typical default model and what are the alternatives.
P6, L189-191: Provide a short explanation how this is done or where this is described.
P8, L237: Since “Performance”, “Processes” and “Teleconnections” are not really metrics, I would suggest to rewrite the sentence as follows: “The ENSO metrics used to assess/evaluate the models are divided into three categories: ………..”.
P16-18: Summary section is too lengthy and not really summarizing what has been presented in the manuscript. Some part of this should rather be part of the discussion section. Further, there should be a conclusion section e.g. stating clearly what is the gain for the community of this software package. Has it already successful applied for CMIP etc.
Figure 1 caption: Add in the figure caption what is shown in the boxes (thus, add that the separation of RMSE by season is shown there)
Technical corrections:
P4, L108: Add “assessment” or “model” after CMIP.
P8, L247: Abbreviation “ITCZ” has not been introduced.
P9, L288: Abbreviation ”DOE” has not been introduced.
P11, L344: Abbreviation GoG has not been introduced. It is given in the figure caption
of figure 7, but not in the text at L338: Should be done there, too.
P12, L401: Although the abbreviations “WGNE” and “WGCM” have been introduced, I would suggest to repeat it here.
P12, L 466: Abbreviation “ESGF” has not been introduced.
P15, L493: …….Section 3.3 are respectively -> Section 3.3., respectively, are
P16, L542: use parenthesis instead of brackets
P18, L605: check sentence.
P41, Figure 6 caption, L1293: units should be given in an upright font.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC1 - AC1: 'Reply on RC1', Jiwoo Lee, 09 Mar 2024
-
RC2: 'Comment on egusphere-2023-2720', Anonymous Referee #2, 18 Dec 2023
COMMENTS TO THE AUTHOR(S)
A comprehensive and objective evaluation of Earth System Models (ESMs) is essential to understand the strengths and weaknesses of individual ESMs and to provide a basis for model improvement. This paper provides an overview of the PCMDI Metric Package (PMP) Version 3, which provides a "quick-look" objective comparison of ESMs to each other and to observations. The purpose, flow, and explanation of the present paper were well written in a concise and easy to understand manner. I expect that the PMP package will be more useful in the context of the CMIP7 project and subsequent phases. I have only a few suggestions before the publication of this paper in the GMD journal.
General suggestion:
I am so curious about how to install the PMP package on my local computer as a user, and what are the requirements for a successful installation. This is the most basic and critical question that readers may have. While the author provides a GitHub link for the related information, I believe it is essential to have a dedicated section in this paper with a brief explanation. This will serve as a catalyst for more readers to become actual PMP users.
Minor comments:
Lines 101-102: I am just wondering if the PMP can be used to evaluate the regional climate models that participated in the RCM project, such as CORDEX. The authors could briefly discuss this or any related future plans (if the authors have any) in the discussion section.
Figure 1. Are all model grids (or just land?) used in the evaluation? It would be helpful to indicate this in the figure caption.
For Abbreviation:
Line 37 (Abstract): Provide the PCMDI's full name.
Line 247. Provide the ITCZ's full name.
Line 288. Provide the DOE's full name.
Line 344. Provide the GoG's full name.
Line 466. Provide the ESGF's full name.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC2 - AC2: 'Reply on RC2', Jiwoo Lee, 09 Mar 2024
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2023-2720', Anonymous Referee #1, 04 Dec 2023
In their manuscript, Lee et al. present the PCDMI Metrics Package (PMP) version 3, which is an open-source Python software package that provides tools for comparisons of ESMs with each other as well as for comparisons of ESMs with observations. A large range of atmospheric processes can be assessed with this software package. Since this package has been developed within the CMIP comparison projects the PMP results can be produced for all model simulations contributing to CMIP6 and earlier CMIP phases.
The manuscript is generally well written and the package useful for the scientific community, but I have several comments that should be taken into account before publication in GMD. Generally, the authors should take care that their message comes better through. At the moment, I have the feeling that there is a lot written, but what is the take home message? Is this a unique package or are there other, similar packages available. What is new or unique for your specific software package?
General comments:
- This is a quite extensive overview and I was wondering how a new user should get started with PMP. I saw from the links you provided that everything is quite well documented, however, this is so much information at once that I think that a new user will have really trouble getting started. Is there e.g. a documentation that could be downloaded as a single pdf file or does one really spend hours reading everything on the computer screen?
- How can a user install PMP on a computer? What requirements are needed? You provide a link to your github repository, where this information is provided, but I think this information should also be provided in the manuscript itself.
- A general questions I was wondering about is, if you are offering workshops for training of new users?
- In the abstract you state what you will discuss the history up to date, recent updates and future directions. The future directions are discussed to some part in Section 6, but I could not find any information or discussion on the history of PMP up to date or what the updates between former and this version were.
- Generally, all sections seem to be a bit too lengthy and to my opinion not really coming to the point. Best example is the summary and future directions section. Although you have a discussion section, you provide a quite long summary and future directions section without really summarizing what you have presented.
- Is this the only tool for analyzing CMIP data or have there also other tools been developed?
Specific comments:
P2, L47-48: When was version 1 developed (published)? What are the major changes/new developments you are presenting here?
P3, L82: Also here you should clearly state when the first version was developed. Has this version been somewhere documented/published? Or has this version just been provided to the CMIP community?
P4, L86: Here, some examples should be given. Which performance metrics or statistical measures are used?
P4, L101: Also here, add some examples.
P4, L102: Why only a subset? Specify.
P4, L102: What do you mean with CMIP class models? Models that participate in CMIP or the kind of models participating in CMIP?
P4, L109: References? Are there any publications?
P6, L170: The following subsections are rather the “processes” that can be assessed with PMP than metrics. The statistics you are using are the metrics for each of this processes.
P6, L174: “well-established statistics” -> You should more clearly write here which statistics are used (at least some examples should be given).
P6, L178: Provide here a typical example. What is a typical default model and what are the alternatives.
P6, L189-191: Provide a short explanation how this is done or where this is described.
P8, L237: Since “Performance”, “Processes” and “Teleconnections” are not really metrics, I would suggest to rewrite the sentence as follows: “The ENSO metrics used to assess/evaluate the models are divided into three categories: ………..”.
P16-18: Summary section is too lengthy and not really summarizing what has been presented in the manuscript. Some part of this should rather be part of the discussion section. Further, there should be a conclusion section e.g. stating clearly what is the gain for the community of this software package. Has it already successful applied for CMIP etc.
Figure 1 caption: Add in the figure caption what is shown in the boxes (thus, add that the separation of RMSE by season is shown there)
Technical corrections:
P4, L108: Add “assessment” or “model” after CMIP.
P8, L247: Abbreviation “ITCZ” has not been introduced.
P9, L288: Abbreviation ”DOE” has not been introduced.
P11, L344: Abbreviation GoG has not been introduced. It is given in the figure caption
of figure 7, but not in the text at L338: Should be done there, too.
P12, L401: Although the abbreviations “WGNE” and “WGCM” have been introduced, I would suggest to repeat it here.
P12, L 466: Abbreviation “ESGF” has not been introduced.
P15, L493: …….Section 3.3 are respectively -> Section 3.3., respectively, are
P16, L542: use parenthesis instead of brackets
P18, L605: check sentence.
P41, Figure 6 caption, L1293: units should be given in an upright font.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC1 - AC1: 'Reply on RC1', Jiwoo Lee, 09 Mar 2024
-
RC2: 'Comment on egusphere-2023-2720', Anonymous Referee #2, 18 Dec 2023
COMMENTS TO THE AUTHOR(S)
A comprehensive and objective evaluation of Earth System Models (ESMs) is essential to understand the strengths and weaknesses of individual ESMs and to provide a basis for model improvement. This paper provides an overview of the PCMDI Metric Package (PMP) Version 3, which provides a "quick-look" objective comparison of ESMs to each other and to observations. The purpose, flow, and explanation of the present paper were well written in a concise and easy to understand manner. I expect that the PMP package will be more useful in the context of the CMIP7 project and subsequent phases. I have only a few suggestions before the publication of this paper in the GMD journal.
General suggestion:
I am so curious about how to install the PMP package on my local computer as a user, and what are the requirements for a successful installation. This is the most basic and critical question that readers may have. While the author provides a GitHub link for the related information, I believe it is essential to have a dedicated section in this paper with a brief explanation. This will serve as a catalyst for more readers to become actual PMP users.
Minor comments:
Lines 101-102: I am just wondering if the PMP can be used to evaluate the regional climate models that participated in the RCM project, such as CORDEX. The authors could briefly discuss this or any related future plans (if the authors have any) in the discussion section.
Figure 1. Are all model grids (or just land?) used in the evaluation? It would be helpful to indicate this in the figure caption.
For Abbreviation:
Line 37 (Abstract): Provide the PCMDI's full name.
Line 247. Provide the ITCZ's full name.
Line 288. Provide the DOE's full name.
Line 344. Provide the GoG's full name.
Line 466. Provide the ESGF's full name.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC2 - AC2: 'Reply on RC2', Jiwoo Lee, 09 Mar 2024
Peer review completion
Journal article(s) based on this preprint
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 324 | 186 | 26 | 536 | 61 | 22 | 17 |
- HTML: 324
- PDF: 186
- XML: 26
- Total: 536
- Supplement: 61
- BibTeX: 22
- EndNote: 17
Viewed (geographical distribution)
| Country | # | Views | % |
|---|---|---|---|
| United States of America | 1 | 261 | 50 |
| China | 2 | 58 | 11 |
| Germany | 3 | 29 | 5 |
| Australia | 4 | 20 | 3 |
| France | 5 | 20 | 3 |
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
- 261
Cited
2 citations as recorded by crossref.
Peter J. Gleckler
Min-Seop Ahn
Ana Ordonez
Paul A. Ullrich
Kenneth R. Sperber
Karl E. Taylor
Yann Y. Planton
Eric Guilyardi
Paul Durack
Celine Bonfils
Mark D. Zelinka
Li-Wei Chao
Bo Dong
Charles Doutriaux
Chengzhu Zhang
Tom Vo
Jason Boutte
Michael F. Wehner
Angeline G. Pendergrass
Daehyun Kim
Andrew T. Wittenberg
John Krasting
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(3528 KB) - Metadata XML
-
Supplement
(1004 KB) - BibTeX
- EndNote
- Final revised paper