Objective Evaluation of Earth System Models: PCMDI Metrics Package (PMP) version 3

Lee, Jiwoo; Gleckler, Peter J.; Ahn, Min-Seop; Ordonez, Ana; Ullrich, Paul A.; Sperber, Kenneth R.; Taylor, Karl E.; Planton, Yann Y.; Guilyardi, Eric; Durack, Paul; Bonfils, Celine; Zelinka, Mark D.; Chao, Li-Wei; Dong, Bo; Doutriaux, Charles; Zhang, Chengzhu; Vo, Tom; Boutte, Jason; Wehner, Michael F.; Pendergrass, Angeline G.; Kim, Daehyun; Xue, Zeyu; Wittenberg, Andrew T.; Krasting, John

doi:https://doi.org/10.5194/egusphere-2023-2720

Preprints

https://doi.org/10.5194/egusphere-2023-2720

Preprints

24 Nov 2023

| 24 Nov 2023

Objective Evaluation of Earth System Models: PCMDI Metrics Package (PMP) version 3

Jiwoo Lee, Peter J. Gleckler, Min-Seop Ahn, Ana Ordonez, Paul A. Ullrich, Kenneth R. Sperber, Karl E. Taylor, Yann Y. Planton, Eric Guilyardi, Paul Durack, Celine Bonfils, Mark D. Zelinka, Li-Wei Chao, Bo Dong, Charles Doutriaux, Chengzhu Zhang, Tom Vo, Jason Boutte, Michael F. Wehner, Angeline G. Pendergrass, Daehyun Kim, Zeyu Xue, Andrew T. Wittenberg, and John Krasting

Abstract. Systematic, routine, and comprehensive evaluation of Earth System Models (ESMs) facilitates benchmarking improvement across model generations and identifying the strengths and weaknesses of different model configurations. By gauging the consistency between models and observations, this endeavor is becoming increasingly necessary to objectively synthesize thousands of simulations contributed to the Coupled Model Intercomparison Project (CMIP) to date. The PCMDI Metrics Package (PMP) is an open-source Python software package that provides "quick-look" objective comparisons of ESMs with one another and with observations. The comparisons include metrics of large- to global-scale climatologies, tropical inter-annual and intra-seasonal variability modes such as El Niño-Southern Oscillation (ENSO) and Madden-Julian Oscillation (MJO), extratropical modes of variability, regional monsoons, cloud radiative feedbacks, and high-frequency characteristics of simulated precipitation, including extremes. The PMP results are produced in the context of all model simulations contributed to CMIP6 and earlier CMIP phases. An important priority of the PMP is to document evaluation statistics for all Historical and AMIP simulations submitted to recent phases of CMIP, providing version-controlled information for all data sets and software packages being used. Among other purposes, this also enables modeling groups to assess performance changes during the ESM development cycle in the context of the error distribution of the multi-model ensemble. In this paper, we present an overview of the PMP including its history to date, capabilities, recent updates, and future direction.

How to cite. Lee, J., Gleckler, P. J., Ahn, M.-S., Ordonez, A., Ullrich, P. A., Sperber, K. R., Taylor, K. E., Planton, Y. Y., Guilyardi, E., Durack, P., Bonfils, C., Zelinka, M. D., Chao, L.-W., Dong, B., Doutriaux, C., Zhang, C., Vo, T., Boutte, J., Wehner, M. F., Pendergrass, A. G., Kim, D., Xue, Z., Wittenberg, A. T., and Krasting, J.: Objective Evaluation of Earth System Models: PCMDI Metrics Package (PMP) version 3, EGUsphere [preprint], https://doi.org/10.5194/egusphere-2023-2720, 2023.

Received: 16 Nov 2023 – Discussion started: 24 Nov 2023

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.

Download & links

Preprint (PDF, 3528 KB)

Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
Preprint (3528 KB)

Supplement (1004 KB)

Download & links

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Journal article(s) based on this preprint

15 May 2024

Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3

Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024,https://doi.org/10.5194/gmd-17-3919-2024, 2024

Short summary

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2023-2720', Anonymous Referee #1, 04 Dec 2023
In their manuscript, Lee et al. present the PCDMI Metrics Package (PMP) version 3, which is an open-source Python software package that provides tools for comparisons of ESMs with each other as well as for comparisons of ESMs with observations. A large range of atmospheric processes can be assessed with this software package. Since this package has been developed within the CMIP comparison projects the PMP results can be produced for all model simulations contributing to CMIP6 and earlier CMIP phases.
The manuscript is generally well written and the package useful for the scientific community, but I have several comments that should be taken into account before publication in GMD. Generally, the authors should take care that their message comes better through. At the moment, I have the feeling that there is a lot written, but what is the take home message? Is this a unique package or are there other, similar packages available. What is new or unique for your specific software package?
General comments:
This is a quite extensive overview and I was wondering how a new user should get started with PMP. I saw from the links you provided that everything is quite well documented, however, this is so much information at once that I think that a new user will have really trouble getting started. Is there e.g. a documentation that could be downloaded as a single pdf file or does one really spend hours reading everything on the computer screen?

How can a user install PMP on a computer? What requirements are needed? You provide a link to your github repository, where this information is provided, but I think this information should also be provided in the manuscript itself.

A general questions I was wondering about is, if you are offering workshops for training of new users?

In the abstract you state what you will discuss the history up to date, recent updates and future directions. The future directions are discussed to some part in Section 6, but I could not find any information or discussion on the history of PMP up to date or what the updates between former and this version were.

Generally, all sections seem to be a bit too lengthy and to my opinion not really coming to the point. Best example is the summary and future directions section. Although you have a discussion section, you provide a quite long summary and future directions section without really summarizing what you have presented.

Is this the only tool for analyzing CMIP data or have there also other tools been developed?

Specific comments:
P2, L47-48: When was version 1 developed (published)? What are the major changes/new developments you are presenting here?
P3, L82: Also here you should clearly state when the first version was developed. Has this version been somewhere documented/published? Or has this version just been provided to the CMIP community?
P4, L86: Here, some examples should be given. Which performance metrics or statistical measures are used?
P4, L101: Also here, add some examples.
P4, L102: Why only a subset? Specify.
P4, L102: What do you mean with CMIP class models? Models that participate in CMIP or the kind of models participating in CMIP?
P4, L109: References? Are there any publications?
P6, L170: The following subsections are rather the “processes” that can be assessed with PMP than metrics. The statistics you are using are the metrics for each of this processes.
P6, L174: “well-established statistics” -> You should more clearly write here which statistics are used (at least some examples should be given).
P6, L178: Provide here a typical example. What is a typical default model and what are the alternatives.
P6, L189-191: Provide a short explanation how this is done or where this is described.
P8, L237: Since “Performance”, “Processes” and “Teleconnections” are not really metrics, I would suggest to rewrite the sentence as follows: “The ENSO metrics used to assess/evaluate the models are divided into three categories: ………..”.
P16-18: Summary section is too lengthy and not really summarizing what has been presented in the manuscript. Some part of this should rather be part of the discussion section. Further, there should be a conclusion section e.g. stating clearly what is the gain for the community of this software package. Has it already successful applied for CMIP etc.
Figure 1 caption: Add in the figure caption what is shown in the boxes (thus, add that the separation of RMSE by season is shown there)
Technical corrections:
P4, L108: Add “assessment” or “model” after CMIP.
P8, L247: Abbreviation “ITCZ” has not been introduced.
P9, L288: Abbreviation ”DOE” has not been introduced.
P11, L344: Abbreviation GoG has not been introduced. It is given in the figure caption
of figure 7, but not in the text at L338: Should be done there, too.
P12, L401: Although the abbreviations “WGNE” and “WGCM” have been introduced, I would suggest to repeat it here.
P12, L 466: Abbreviation “ESGF” has not been introduced.
P15, L493: …….Section 3.3 are respectively -> Section 3.3., respectively, are
P16, L542: use parenthesis instead of brackets
P18, L605: check sentence.
P41, Figure 6 caption, L1293: units should be given in an upright font.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC1
- AC1: 'Reply on RC1', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC1
RC2:
'Comment on egusphere-2023-2720', Anonymous Referee #2, 18 Dec 2023

COMMENTS TO THE AUTHOR(S)
A comprehensive and objective evaluation of Earth System Models (ESMs) is essential to understand the strengths and weaknesses of individual ESMs and to provide a basis for model improvement. This paper provides an overview of the PCMDI Metric Package (PMP) Version 3, which provides a "quick-look" objective comparison of ESMs to each other and to observations. The purpose, flow, and explanation of the present paper were well written in a concise and easy to understand manner. I expect that the PMP package will be more useful in the context of the CMIP7 project and subsequent phases. I have only a few suggestions before the publication of this paper in the GMD journal.

General suggestion:
I am so curious about how to install the PMP package on my local computer as a user, and what are the requirements for a successful installation. This is the most basic and critical question that readers may have. While the author provides a GitHub link for the related information, I believe it is essential to have a dedicated section in this paper with a brief explanation. This will serve as a catalyst for more readers to become actual PMP users.

Minor comments:
Lines 101-102: I am just wondering if the PMP can be used to evaluate the regional climate models that participated in the RCM project, such as CORDEX. The authors could briefly discuss this or any related future plans (if the authors have any) in the discussion section.
Figure 1. Are all model grids (or just land?) used in the evaluation? It would be helpful to indicate this in the figure caption.

For Abbreviation:
Line 37 (Abstract): Provide the PCMDI's full name.
Line 247. Provide the ITCZ's full name.
Line 288. Provide the DOE's full name.
Line 344. Provide the GoG's full name.
Line 466. Provide the ESGF's full name.

Citation: https://doi.org/10.5194/egusphere-2023-2720-RC2
- AC2: 'Reply on RC2', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC2

Interactive discussion

Status: closed

RC1:
'Comment on egusphere-2023-2720', Anonymous Referee #1, 04 Dec 2023
In their manuscript, Lee et al. present the PCDMI Metrics Package (PMP) version 3, which is an open-source Python software package that provides tools for comparisons of ESMs with each other as well as for comparisons of ESMs with observations. A large range of atmospheric processes can be assessed with this software package. Since this package has been developed within the CMIP comparison projects the PMP results can be produced for all model simulations contributing to CMIP6 and earlier CMIP phases.
The manuscript is generally well written and the package useful for the scientific community, but I have several comments that should be taken into account before publication in GMD. Generally, the authors should take care that their message comes better through. At the moment, I have the feeling that there is a lot written, but what is the take home message? Is this a unique package or are there other, similar packages available. What is new or unique for your specific software package?
General comments:
This is a quite extensive overview and I was wondering how a new user should get started with PMP. I saw from the links you provided that everything is quite well documented, however, this is so much information at once that I think that a new user will have really trouble getting started. Is there e.g. a documentation that could be downloaded as a single pdf file or does one really spend hours reading everything on the computer screen?

How can a user install PMP on a computer? What requirements are needed? You provide a link to your github repository, where this information is provided, but I think this information should also be provided in the manuscript itself.

A general questions I was wondering about is, if you are offering workshops for training of new users?

In the abstract you state what you will discuss the history up to date, recent updates and future directions. The future directions are discussed to some part in Section 6, but I could not find any information or discussion on the history of PMP up to date or what the updates between former and this version were.

Generally, all sections seem to be a bit too lengthy and to my opinion not really coming to the point. Best example is the summary and future directions section. Although you have a discussion section, you provide a quite long summary and future directions section without really summarizing what you have presented.

Is this the only tool for analyzing CMIP data or have there also other tools been developed?

Specific comments:
P2, L47-48: When was version 1 developed (published)? What are the major changes/new developments you are presenting here?
P3, L82: Also here you should clearly state when the first version was developed. Has this version been somewhere documented/published? Or has this version just been provided to the CMIP community?
P4, L86: Here, some examples should be given. Which performance metrics or statistical measures are used?
P4, L101: Also here, add some examples.
P4, L102: Why only a subset? Specify.
P4, L102: What do you mean with CMIP class models? Models that participate in CMIP or the kind of models participating in CMIP?
P4, L109: References? Are there any publications?
P6, L170: The following subsections are rather the “processes” that can be assessed with PMP than metrics. The statistics you are using are the metrics for each of this processes.
P6, L174: “well-established statistics” -> You should more clearly write here which statistics are used (at least some examples should be given).
P6, L178: Provide here a typical example. What is a typical default model and what are the alternatives.
P6, L189-191: Provide a short explanation how this is done or where this is described.
P8, L237: Since “Performance”, “Processes” and “Teleconnections” are not really metrics, I would suggest to rewrite the sentence as follows: “The ENSO metrics used to assess/evaluate the models are divided into three categories: ………..”.
P16-18: Summary section is too lengthy and not really summarizing what has been presented in the manuscript. Some part of this should rather be part of the discussion section. Further, there should be a conclusion section e.g. stating clearly what is the gain for the community of this software package. Has it already successful applied for CMIP etc.
Figure 1 caption: Add in the figure caption what is shown in the boxes (thus, add that the separation of RMSE by season is shown there)
Technical corrections:
P4, L108: Add “assessment” or “model” after CMIP.
P8, L247: Abbreviation “ITCZ” has not been introduced.
P9, L288: Abbreviation ”DOE” has not been introduced.
P11, L344: Abbreviation GoG has not been introduced. It is given in the figure caption
of figure 7, but not in the text at L338: Should be done there, too.
P12, L401: Although the abbreviations “WGNE” and “WGCM” have been introduced, I would suggest to repeat it here.
P12, L 466: Abbreviation “ESGF” has not been introduced.
P15, L493: …….Section 3.3 are respectively -> Section 3.3., respectively, are
P16, L542: use parenthesis instead of brackets
P18, L605: check sentence.
P41, Figure 6 caption, L1293: units should be given in an upright font.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC1
- AC1: 'Reply on RC1', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC1
RC2:
'Comment on egusphere-2023-2720', Anonymous Referee #2, 18 Dec 2023

COMMENTS TO THE AUTHOR(S)
A comprehensive and objective evaluation of Earth System Models (ESMs) is essential to understand the strengths and weaknesses of individual ESMs and to provide a basis for model improvement. This paper provides an overview of the PCMDI Metric Package (PMP) Version 3, which provides a "quick-look" objective comparison of ESMs to each other and to observations. The purpose, flow, and explanation of the present paper were well written in a concise and easy to understand manner. I expect that the PMP package will be more useful in the context of the CMIP7 project and subsequent phases. I have only a few suggestions before the publication of this paper in the GMD journal.

General suggestion:
I am so curious about how to install the PMP package on my local computer as a user, and what are the requirements for a successful installation. This is the most basic and critical question that readers may have. While the author provides a GitHub link for the related information, I believe it is essential to have a dedicated section in this paper with a brief explanation. This will serve as a catalyst for more readers to become actual PMP users.

Minor comments:
Lines 101-102: I am just wondering if the PMP can be used to evaluate the regional climate models that participated in the RCM project, such as CORDEX. The authors could briefly discuss this or any related future plans (if the authors have any) in the discussion section.
Figure 1. Are all model grids (or just land?) used in the evaluation? It would be helpful to indicate this in the figure caption.

For Abbreviation:
Line 37 (Abstract): Provide the PCMDI's full name.
Line 247. Provide the ITCZ's full name.
Line 288. Provide the DOE's full name.
Line 344. Provide the GoG's full name.
Line 466. Provide the ESGF's full name.

Citation: https://doi.org/10.5194/egusphere-2023-2720-RC2
- AC2: 'Reply on RC2', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC2

Peer review completion

AR: Author's response | RR: Referee report | ED: Editor decision | EF: Editorial file upload

AR by Jiwoo Lee on behalf of the Authors (09 Mar 2024) Author's response Author's tracked changes Manuscript

ED: Referee Nomination & Report Request started (13 Mar 2024) by Lele Shu

RR by Anonymous Referee #2 (15 Mar 2024)

RR by Anonymous Referee #1 (21 Mar 2024)

Suggestions for revision or reasons for rejection

The revised version of the manuscript has significantly improved. I have only some technical corrections that should be considered before publication.

P2, L42: “including extremes”. What exactly do you mean with extremes? What kind of extremes? Of precipitation? Give an example.

P2, L42: The PMP results -> The PMP comparison results

P2, L42: “in the context of all model simulations contributing to….” Not clear, please rephrase. Instead of writing ”in the context” I would rather write “using”.

P2, L43: “priority”? I would rather say “objective” is here the correct term.

P2, L45: process? Rather use singular than plural?

P3, L53: “……has been an exponential growth of data size….” Mentioning this here makes no sense. You further mention this some lines below ones again, where it makes more sense. Thus, I would suggest to remove this text part here.

P4, L118 and 119: Section should be abbreviated as Sect. except at the begin of the Sentence (see manuscript preparation guidelines). This should also be corrected throughout the manuscript.

P5, L131: “…protocol constrains the simulation with….” Is here not “prescribes” the correct term rather than “constrains”?

P5, L143: priority -> objective

P6, L188 and 189: “We also release ….” and “are also available online” -> this is a repetition and the sentence should be rephrased.

P6, L189 - 190: Add comma after “statistics” and add a comma before include, so that it reads: “The archive of these statistics, stored as JSON files (………), includes……..”.

P6, L191: See -> see

P7, L228: delete “at” before regional scales (?)

P7, L230: these -> the (?)

P10, L323: Write SAM_SON without a dash. Even better it would be if you would skip the abbreviation SON since it has not been introduced and is not in the list of abbreviation.

P11, L375ff: pentads? I do not see any pentads here, but only squares or rectangles.

P13, L435: Figure 8 -> Fig. 8

P13, L450 and P14, L481: Tropics -> tropics

P14, L462: evaluations -> evaluation

P29, L887: reference of Hintze -> remove indent for the first line of the reference.

P30, L893: hydrometeorology -> Hydrometeorology

References in general: Journal names are usually abbreviated (see Copernicus manuscript preparation guidelines)

Hide

ED: Publish subject to minor revisions (review by editor) (28 Mar 2024) by Lele Shu

AR by Jiwoo Lee on behalf of the Authors (28 Mar 2024) Author's response Author's tracked changes Manuscript

ED: Publish as is (03 Apr 2024) by Lele Shu

AR by Jiwoo Lee on behalf of the Authors (04 Apr 2024) Manuscript

Journal article(s) based on this preprint

15 May 2024

Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3

Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024,https://doi.org/10.5194/gmd-17-3919-2024, 2024

Short summary

Supplement

https://doi.org/10.5194/egusphere-2023-2720-supplement

Viewed

Total article views: 536 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
324	186	26	536	61	22	17

HTML: 324
PDF: 186
XML: 26
Total: 536
Supplement: 61
BibTeX: 22
EndNote: 17

Views and downloads (calculated since 24 Nov 2023)

Month	HTML	PDF	XML	Total
Nov 2023	41	16	1	58
Dec 2023	95	36	9	140
Jan 2024	38	25	2	65
Feb 2024	20	23	1	44
Mar 2024	65	56	6	127
Apr 2024	47	23	7	77
May 2024	18	7	0	25

Cumulative views and downloads (calculated since 24 Nov 2023)

Month	HTML	PDF	XML	Total
Nov 2023	41	16	1	58
Dec 2023	95	36	9	140
Jan 2024	38	25	2	65
Feb 2024	20	23	1	44
Mar 2024	65	56	6	127
Apr 2024	47	23	7	77
May 2024	18	7	0	25

Viewed (geographical distribution)

Total article views: 522 (including HTML, PDF, and XML) Thereof 522 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 15 May 2024

Download

The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.

Preprint (3528 KB)
Metadata XML

Short summary

We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a comprehensive comparison of Earth System Models (ESMs) with real-world observations. Using diverse metrics evaluating climatology, variability, and extremes simulated in thousands of simulations from the Coupled Model Intercomparison Project (CMIP), PMP aids in benchmarking model improvements across generations. PMP also enables efficient tracking of performance evolutions during ESM developments.


Total:	0
HTML:	0
PDF:	0
XML:	0