Objective Evaluation of Earth System Models: PCMDI Metrics Package (PMP) version 3

Lee, Jiwoo; Gleckler, Peter J.; Ahn, Min-Seop; Ordonez, Ana; Ullrich, Paul A.; Sperber, Kenneth R.; Taylor, Karl E.; Planton, Yann Y.; Guilyardi, Eric; Durack, Paul; Bonfils, Celine; Zelinka, Mark D.; Chao, Li-Wei; Dong, Bo; Doutriaux, Charles; Zhang, Chengzhu; Vo, Tom; Boutte, Jason; Wehner, Michael F.; Pendergrass, Angeline G.; Kim, Daehyun; Xue, Zeyu; Wittenberg, Andrew T.; Krasting, John

doi:https://doi.org/10.5194/egusphere-2023-2720

Preprints

https://doi.org/10.5194/egusphere-2023-2720

Preprints

24 Nov 2023

| 24 Nov 2023

Objective Evaluation of Earth System Models: PCMDI Metrics Package (PMP) version 3

Jiwoo Lee, Peter J. Gleckler, Min-Seop Ahn, Ana Ordonez, Paul A. Ullrich, Kenneth R. Sperber, Karl E. Taylor, Yann Y. Planton, Eric Guilyardi, Paul Durack, Celine Bonfils, Mark D. Zelinka, Li-Wei Chao, Bo Dong, Charles Doutriaux, Chengzhu Zhang, Tom Vo, Jason Boutte, Michael F. Wehner, Angeline G. Pendergrass, Daehyun Kim, Zeyu Xue, Andrew T. Wittenberg, and John Krasting

Abstract. Systematic, routine, and comprehensive evaluation of Earth System Models (ESMs) facilitates benchmarking improvement across model generations and identifying the strengths and weaknesses of different model configurations. By gauging the consistency between models and observations, this endeavor is becoming increasingly necessary to objectively synthesize thousands of simulations contributed to the Coupled Model Intercomparison Project (CMIP) to date. The PCMDI Metrics Package (PMP) is an open-source Python software package that provides "quick-look" objective comparisons of ESMs with one another and with observations. The comparisons include metrics of large- to global-scale climatologies, tropical inter-annual and intra-seasonal variability modes such as El Niño-Southern Oscillation (ENSO) and Madden-Julian Oscillation (MJO), extratropical modes of variability, regional monsoons, cloud radiative feedbacks, and high-frequency characteristics of simulated precipitation, including extremes. The PMP results are produced in the context of all model simulations contributed to CMIP6 and earlier CMIP phases. An important priority of the PMP is to document evaluation statistics for all Historical and AMIP simulations submitted to recent phases of CMIP, providing version-controlled information for all data sets and software packages being used. Among other purposes, this also enables modeling groups to assess performance changes during the ESM development cycle in the context of the error distribution of the multi-model ensemble. In this paper, we present an overview of the PMP including its history to date, capabilities, recent updates, and future direction.

How to cite. Lee, J., Gleckler, P. J., Ahn, M.-S., Ordonez, A., Ullrich, P. A., Sperber, K. R., Taylor, K. E., Planton, Y. Y., Guilyardi, E., Durack, P., Bonfils, C., Zelinka, M. D., Chao, L.-W., Dong, B., Doutriaux, C., Zhang, C., Vo, T., Boutte, J., Wehner, M. F., Pendergrass, A. G., Kim, D., Xue, Z., Wittenberg, A. T., and Krasting, J.: Objective Evaluation of Earth System Models: PCMDI Metrics Package (PMP) version 3, EGUsphere [preprint], https://doi.org/10.5194/egusphere-2023-2720, 2023.

Received: 16 Nov 2023 – Discussion started: 24 Nov 2023

Download & links

Preprint (PDF, 3528 KB)

Supplement (1004 KB)

Download & links

Status: closed

RC1:
'Comment on egusphere-2023-2720', Anonymous Referee #1, 04 Dec 2023
In their manuscript, Lee et al. present the PCDMI Metrics Package (PMP) version 3, which is an open-source Python software package that provides tools for comparisons of ESMs with each other as well as for comparisons of ESMs with observations. A large range of atmospheric processes can be assessed with this software package. Since this package has been developed within the CMIP comparison projects the PMP results can be produced for all model simulations contributing to CMIP6 and earlier CMIP phases.
The manuscript is generally well written and the package useful for the scientific community, but I have several comments that should be taken into account before publication in GMD. Generally, the authors should take care that their message comes better through. At the moment, I have the feeling that there is a lot written, but what is the take home message? Is this a unique package or are there other, similar packages available. What is new or unique for your specific software package?
General comments:
This is a quite extensive overview and I was wondering how a new user should get started with PMP. I saw from the links you provided that everything is quite well documented, however, this is so much information at once that I think that a new user will have really trouble getting started. Is there e.g. a documentation that could be downloaded as a single pdf file or does one really spend hours reading everything on the computer screen?

How can a user install PMP on a computer? What requirements are needed? You provide a link to your github repository, where this information is provided, but I think this information should also be provided in the manuscript itself.

A general questions I was wondering about is, if you are offering workshops for training of new users?

In the abstract you state what you will discuss the history up to date, recent updates and future directions. The future directions are discussed to some part in Section 6, but I could not find any information or discussion on the history of PMP up to date or what the updates between former and this version were.

Generally, all sections seem to be a bit too lengthy and to my opinion not really coming to the point. Best example is the summary and future directions section. Although you have a discussion section, you provide a quite long summary and future directions section without really summarizing what you have presented.

Is this the only tool for analyzing CMIP data or have there also other tools been developed?

Specific comments:
P2, L47-48: When was version 1 developed (published)? What are the major changes/new developments you are presenting here?
P3, L82: Also here you should clearly state when the first version was developed. Has this version been somewhere documented/published? Or has this version just been provided to the CMIP community?
P4, L86: Here, some examples should be given. Which performance metrics or statistical measures are used?
P4, L101: Also here, add some examples.
P4, L102: Why only a subset? Specify.
P4, L102: What do you mean with CMIP class models? Models that participate in CMIP or the kind of models participating in CMIP?
P4, L109: References? Are there any publications?
P6, L170: The following subsections are rather the “processes” that can be assessed with PMP than metrics. The statistics you are using are the metrics for each of this processes.
P6, L174: “well-established statistics” -> You should more clearly write here which statistics are used (at least some examples should be given).
P6, L178: Provide here a typical example. What is a typical default model and what are the alternatives.
P6, L189-191: Provide a short explanation how this is done or where this is described.
P8, L237: Since “Performance”, “Processes” and “Teleconnections” are not really metrics, I would suggest to rewrite the sentence as follows: “The ENSO metrics used to assess/evaluate the models are divided into three categories: ………..”.
P16-18: Summary section is too lengthy and not really summarizing what has been presented in the manuscript. Some part of this should rather be part of the discussion section. Further, there should be a conclusion section e.g. stating clearly what is the gain for the community of this software package. Has it already successful applied for CMIP etc.
Figure 1 caption: Add in the figure caption what is shown in the boxes (thus, add that the separation of RMSE by season is shown there)
Technical corrections:
P4, L108: Add “assessment” or “model” after CMIP.
P8, L247: Abbreviation “ITCZ” has not been introduced.
P9, L288: Abbreviation ”DOE” has not been introduced.
P11, L344: Abbreviation GoG has not been introduced. It is given in the figure caption
of figure 7, but not in the text at L338: Should be done there, too.
P12, L401: Although the abbreviations “WGNE” and “WGCM” have been introduced, I would suggest to repeat it here.
P12, L 466: Abbreviation “ESGF” has not been introduced.
P15, L493: …….Section 3.3 are respectively -> Section 3.3., respectively, are
P16, L542: use parenthesis instead of brackets
P18, L605: check sentence.
P41, Figure 6 caption, L1293: units should be given in an upright font.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC1
- AC1: 'Reply on RC1', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC1
RC2:
'Comment on egusphere-2023-2720', Anonymous Referee #2, 18 Dec 2023

COMMENTS TO THE AUTHOR(S)
A comprehensive and objective evaluation of Earth System Models (ESMs) is essential to understand the strengths and weaknesses of individual ESMs and to provide a basis for model improvement. This paper provides an overview of the PCMDI Metric Package (PMP) Version 3, which provides a "quick-look" objective comparison of ESMs to each other and to observations. The purpose, flow, and explanation of the present paper were well written in a concise and easy to understand manner. I expect that the PMP package will be more useful in the context of the CMIP7 project and subsequent phases. I have only a few suggestions before the publication of this paper in the GMD journal.

General suggestion:
I am so curious about how to install the PMP package on my local computer as a user, and what are the requirements for a successful installation. This is the most basic and critical question that readers may have. While the author provides a GitHub link for the related information, I believe it is essential to have a dedicated section in this paper with a brief explanation. This will serve as a catalyst for more readers to become actual PMP users.

Minor comments:
Lines 101-102: I am just wondering if the PMP can be used to evaluate the regional climate models that participated in the RCM project, such as CORDEX. The authors could briefly discuss this or any related future plans (if the authors have any) in the discussion section.
Figure 1. Are all model grids (or just land?) used in the evaluation? It would be helpful to indicate this in the figure caption.

For Abbreviation:
Line 37 (Abstract): Provide the PCMDI's full name.
Line 247. Provide the ITCZ's full name.
Line 288. Provide the DOE's full name.
Line 344. Provide the GoG's full name.
Line 466. Provide the ESGF's full name.

Citation: https://doi.org/10.5194/egusphere-2023-2720-RC2
- AC2: 'Reply on RC2', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC2

Status: closed

RC1:
'Comment on egusphere-2023-2720', Anonymous Referee #1, 04 Dec 2023
In their manuscript, Lee et al. present the PCDMI Metrics Package (PMP) version 3, which is an open-source Python software package that provides tools for comparisons of ESMs with each other as well as for comparisons of ESMs with observations. A large range of atmospheric processes can be assessed with this software package. Since this package has been developed within the CMIP comparison projects the PMP results can be produced for all model simulations contributing to CMIP6 and earlier CMIP phases.
The manuscript is generally well written and the package useful for the scientific community, but I have several comments that should be taken into account before publication in GMD. Generally, the authors should take care that their message comes better through. At the moment, I have the feeling that there is a lot written, but what is the take home message? Is this a unique package or are there other, similar packages available. What is new or unique for your specific software package?
General comments:
This is a quite extensive overview and I was wondering how a new user should get started with PMP. I saw from the links you provided that everything is quite well documented, however, this is so much information at once that I think that a new user will have really trouble getting started. Is there e.g. a documentation that could be downloaded as a single pdf file or does one really spend hours reading everything on the computer screen?

How can a user install PMP on a computer? What requirements are needed? You provide a link to your github repository, where this information is provided, but I think this information should also be provided in the manuscript itself.

A general questions I was wondering about is, if you are offering workshops for training of new users?

In the abstract you state what you will discuss the history up to date, recent updates and future directions. The future directions are discussed to some part in Section 6, but I could not find any information or discussion on the history of PMP up to date or what the updates between former and this version were.

Generally, all sections seem to be a bit too lengthy and to my opinion not really coming to the point. Best example is the summary and future directions section. Although you have a discussion section, you provide a quite long summary and future directions section without really summarizing what you have presented.

Is this the only tool for analyzing CMIP data or have there also other tools been developed?

Specific comments:
P2, L47-48: When was version 1 developed (published)? What are the major changes/new developments you are presenting here?
P3, L82: Also here you should clearly state when the first version was developed. Has this version been somewhere documented/published? Or has this version just been provided to the CMIP community?
P4, L86: Here, some examples should be given. Which performance metrics or statistical measures are used?
P4, L101: Also here, add some examples.
P4, L102: Why only a subset? Specify.
P4, L102: What do you mean with CMIP class models? Models that participate in CMIP or the kind of models participating in CMIP?
P4, L109: References? Are there any publications?
P6, L170: The following subsections are rather the “processes” that can be assessed with PMP than metrics. The statistics you are using are the metrics for each of this processes.
P6, L174: “well-established statistics” -> You should more clearly write here which statistics are used (at least some examples should be given).
P6, L178: Provide here a typical example. What is a typical default model and what are the alternatives.
P6, L189-191: Provide a short explanation how this is done or where this is described.
P8, L237: Since “Performance”, “Processes” and “Teleconnections” are not really metrics, I would suggest to rewrite the sentence as follows: “The ENSO metrics used to assess/evaluate the models are divided into three categories: ………..”.
P16-18: Summary section is too lengthy and not really summarizing what has been presented in the manuscript. Some part of this should rather be part of the discussion section. Further, there should be a conclusion section e.g. stating clearly what is the gain for the community of this software package. Has it already successful applied for CMIP etc.
Figure 1 caption: Add in the figure caption what is shown in the boxes (thus, add that the separation of RMSE by season is shown there)
Technical corrections:
P4, L108: Add “assessment” or “model” after CMIP.
P8, L247: Abbreviation “ITCZ” has not been introduced.
P9, L288: Abbreviation ”DOE” has not been introduced.
P11, L344: Abbreviation GoG has not been introduced. It is given in the figure caption
of figure 7, but not in the text at L338: Should be done there, too.
P12, L401: Although the abbreviations “WGNE” and “WGCM” have been introduced, I would suggest to repeat it here.
P12, L 466: Abbreviation “ESGF” has not been introduced.
P15, L493: …….Section 3.3 are respectively -> Section 3.3., respectively, are
P16, L542: use parenthesis instead of brackets
P18, L605: check sentence.
P41, Figure 6 caption, L1293: units should be given in an upright font.
Citation: https://doi.org/10.5194/egusphere-2023-2720-RC1
- AC1: 'Reply on RC1', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC1
RC2:
'Comment on egusphere-2023-2720', Anonymous Referee #2, 18 Dec 2023

COMMENTS TO THE AUTHOR(S)
A comprehensive and objective evaluation of Earth System Models (ESMs) is essential to understand the strengths and weaknesses of individual ESMs and to provide a basis for model improvement. This paper provides an overview of the PCMDI Metric Package (PMP) Version 3, which provides a "quick-look" objective comparison of ESMs to each other and to observations. The purpose, flow, and explanation of the present paper were well written in a concise and easy to understand manner. I expect that the PMP package will be more useful in the context of the CMIP7 project and subsequent phases. I have only a few suggestions before the publication of this paper in the GMD journal.

General suggestion:
I am so curious about how to install the PMP package on my local computer as a user, and what are the requirements for a successful installation. This is the most basic and critical question that readers may have. While the author provides a GitHub link for the related information, I believe it is essential to have a dedicated section in this paper with a brief explanation. This will serve as a catalyst for more readers to become actual PMP users.

Minor comments:
Lines 101-102: I am just wondering if the PMP can be used to evaluate the regional climate models that participated in the RCM project, such as CORDEX. The authors could briefly discuss this or any related future plans (if the authors have any) in the discussion section.
Figure 1. Are all model grids (or just land?) used in the evaluation? It would be helpful to indicate this in the figure caption.

For Abbreviation:
Line 37 (Abstract): Provide the PCMDI's full name.
Line 247. Provide the ITCZ's full name.
Line 288. Provide the DOE's full name.
Line 344. Provide the GoG's full name.
Line 466. Provide the ESGF's full name.

Citation: https://doi.org/10.5194/egusphere-2023-2720-RC2
- AC2: 'Reply on RC2', Jiwoo Lee, 09 Mar 2024
  
  We appreciate the reviewer’s time for providing thoughtful and constructive comments. Please find our point-by-point response in the attached file.
  
  Citation: https://doi.org/10.5194/egusphere-2023-2720-AC2

Supplement

https://doi.org/10.5194/egusphere-2023-2720-supplement

Viewed

Total article views: 501 (including HTML, PDF, and XML)

HTML	PDF	XML	Total	Supplement	BibTeX	EndNote
301	177	23	501	56	19	13

HTML: 301
PDF: 177
XML: 23
Total: 501
Supplement: 56
BibTeX: 19
EndNote: 13

Views and downloads (calculated since 24 Nov 2023)

Month	HTML	PDF	XML	Total
Nov 2023	41	16	1	58
Dec 2023	95	36	9	140
Jan 2024	38	25	2	65
Feb 2024	20	23	1	44
Mar 2024	65	56	6	127
Apr 2024	42	21	4	67

Cumulative views and downloads (calculated since 24 Nov 2023)

Month	HTML	PDF	XML	Total
Nov 2023	41	16	1	58
Dec 2023	95	36	9	140
Jan 2024	38	25	2	65
Feb 2024	20	23	1	44
Mar 2024	65	56	6	127
Apr 2024	42	21	4	67

Viewed (geographical distribution)

Total article views: 487 (including HTML, PDF, and XML) Thereof 487 with geography defined and 0 with unknown origin.

Country	#	Views	%

Latest update: 27 Apr 2024

Short summary

We introduce an open-source software, the PCMDI Metrics Package (PMP), developed for a comprehensive comparison of Earth System Models (ESMs) with real-world observations. Using diverse metrics evaluating climatology, variability, and extremes simulated in thousands of simulations from the Coupled Model Intercomparison Project (CMIP), PMP aids in benchmarking model improvements across generations. PMP also enables efficient tracking of performance evolutions during ESM developments.


Total:	0
HTML:	0
PDF:	0
XML:	0