the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Evaluating Precipitation Distributions at Regional Scales: A Benchmarking Framework and Application to CMIP 5 and 6 Models
Min-Seop Ahn
Paul A. Ullrich
Peter J. Gleckler
Jiwoo Lee
Ana C. Ordonez
Angeline G. Pendergrass
Abstract. A framework for quantifying precipitation distributions at regional scales is presented and applied to CMIP 5 and 6 models. We employ the IPCC AR6 climate reference regions over land and propose refinements to the oceanic regions based on the homogeneity of precipitation distribution characteristics. The homogeneous regions are identified as heavy, moderate, and light precipitating areas by K-means clustering of IMERG precipitation frequency and amount distributions. With the global domain partitioned into 62 regions, including 46 land and 16 ocean regions, we apply 10 established precipitation distribution metrics. The collection includes metrics focused on the maximum peak, lower 10th percentile, and upper 90th percentile in precipitation amount and frequency distributions, the similarity between observed and modeled frequency distributions, an unevenness measure based on cumulative amount, average total intensity on all days with precipitation, and number of precipitating days each year. We apply our framework to 25 CMIP5 and 41 CMIP6 models, and 6 observation-based products of daily precipitation. Our results indicate that many CMIP 5 and 6 models substantially overestimate the observed light precipitation amount and frequency as well as the number of precipitating days, especially over mid-latitude regions outside of some land regions in the Americas and Eurasia. Improvement from CMIP 5 to 6 is shown in some regions, especially in mid-latitude regions, but it is not evident globally, and over the tropics most metrics point toward over degradation.
- Preprint
(4544 KB) -
Supplement
(1156 KB) - BibTeX
- EndNote
Min-Seop Ahn et al.
Status: final response (author comments only)
-
CC1: 'Comment on egusphere-2022-1106', Hong Yan, 24 Feb 2023
The paper by Sophie et al. aims to build a framework at regional scale and then apply to cmip models to evaluate the simulated precipitation distributions. Employing IPCC AR6 climate reference regions over land and oceanic regions modified by homogeneity, together with 10 established precipitation distribution metrics, the author meticulously assessed precipitation distributions simulated by CMIP 5 and 6 models with 6 observation-based products. The paper proved the robustness of homogeneity within reference regions by considering the different resolutions and seasons, as well as fully evaluate the model simulations against multiple observations. I find the paper quite interesting and in line with the scope of EGUsphere. But there are some problems, which must be solved before it is considered for publication. The details are as follows:
- In abstract, the author introduced the methods and results clearly. But the background and significance of this paper is not expound sufficiently. The author need to highlight this paper's background information and innovative contributions.
- Line 175-178: Except for frequency and amount distributions, why not take the cumulative amount fraction as one of the variables in clustering the similar precipitation distribution region? Please explain them briefly.
- Line 182-183: What are the criteria for defining heavy, moderate and light rain regions?
- Line 201-227：I suggest these contents should be simplified.
- In part 4.1, the homogeneity for precipitation distributions within the regions were fully assessed. I suggest evaluating the heterogeneity between regions with different precipitation distribution briefly.
- I suggest the text in part 4.2 and conclusion need to be simplified.
Two minor comments:
- Line 107: The abbreviation ECMWF should be written in European Center for Medium-Range Weather Forecasts.
- Unify the format of references’ websites
Citation: https://doi.org/10.5194/egusphere-2022-1106-CC1 -
AC1: 'Reply on CC1', Min-Seop Ahn, 20 Apr 2023
The paper by Sophie et al. aims to build a framework at regional scale and then apply to cmip models to evaluate the simulated precipitation distributions. Employing IPCC AR6 climate reference regions over land and oceanic regions modified by homogeneity, together with 10 established precipitation distribution metrics, the author meticulously assessed precipitation distributions simulated by CMIP 5 and 6 models with 6 observation-based products. The paper proved the robustness of homogeneity within reference regions by considering the different resolutions and seasons, as well as fully evaluate the model simulations against multiple observations. I find the paper quite interesting and in line with the scope of EGUsphere. But there are some problems, which must be solved before it is considered for publication. The details are as follows:
We thank the reviewer for the helpful and constructive comments. Please find our point-by-point response below. Note that we will upload our revised manuscript after we address all referees’ comments. Thus we have answered this community comments by describing how we will revise the manuscript.
1. In abstract, the author introduced the methods and results clearly. But the background and significance of this paper is not expound sufficiently. The author need to highlight this paper's background information and innovative contributions.
We will revise the abstract to add background information at the first of the abstract. Below is the anticipated revision:
"As the resolution of global Earth system models increases, regional scale evaluation is becoming ever more important. This study presents a framework for quantifying precipitation distributions at regional scales and applies it to evaluate CMIP 5 and 6 models.”
2. Line 175-178: Except for frequency and amount distributions, why not take the cumulative amount fraction as one of the variables in clustering the similar precipitation distribution region? Please explain them briefly.
Thank you for the question which has prompted us to test the precipitation clustering. The clustering result with frequency and amount distributions (Fig. R1a) is very similar to the result with the additional variable of cumulative amount fraction (Fig. R1b). This indicates that our region definition is not meaningfully affected by adding the cumulative amount fraction. When we add the cumulative amount fraction to the clustering, however, the result tends towards a noisier pattern. Also, the current version (Fig. R1a) is more similar to the mean state pattern (Fig. 2a), notably over eastern North America and the southern Indian Ocean. This reinforces our choice to use the clustering result with frequency and amount distributions, but we will discuss this consideration in the revised manuscript. Below is the anticipated addition to the manuscript:
“Note that the clustering result with frequency and amount distributions is not significantly altered if we incorporate cumulative amount fraction. However, the inclusion of the cumulative amount fraction to the clustering yields a slightly noisier pattern, and thus we have chosen to use the clustering result only with frequency and amount distributions.”
Figure R1. Spatial patterns of clustering for heavy, moderate, and light precipitating regions by K-means clustering with a) amount and frequency distributions and b) amount, frequency, and cumulative amount fraction distributions.
3. Line 182-183: What are the criteria for defining heavy, moderate and light rain regions?
We did not use any criteria of precipitation intensity for defining the precipitation regions, rather, it is one of the outcomes of the K-means clustering. The K-means clustering algorithm defines the homogeneity regions by iterating to minimize the sum of distances between the cluster centroid and each cluster member. The only parameter for K-means clustering is the number of desired clusters. Here we use 3 for defining heavy, moderate, and light rain regions. But your query has convinced us to clarify this point. Below is the anticipated revision:
“K-means clustering is an unsupervised machine learning algorithm that separates characteristics of a dataset into a given number of clusters without any inputs of criteria values. This method has been widely used because it is faster and simpler than other methods. Here, we use 3 clusters to define heavy, moderate, and light precipitation regions.”
4. Line 201-227：I suggest these contents should be simplified.
We will simplify this part in the revised manuscript.
5. In part 4.1, the homogeneity for precipitation distributions within the regions were fully assessed. I suggest evaluating the heterogeneity between regions with different precipitation distribution briefly.
We think that the homogeneity within regions and the heterogeneity between regions implicitly reveal opposing information. As our objective is defining homogeneity regions, we would propose that showing homogeneity within regions would be sufficient to justify our defining regions.
6. I suggest the text in part 4.2 and conclusion need to be simplified.
We will simplify section 4.2 and the conclusion in the revised manuscript.
Two minor comments:
1. Line 107: The abbreviation ECMWF should be written in European Center for Medium-Range Weather Forecasts.
We will correct this in the revised manuscript.
2. Unify the format of references’ websites
We will unify the format of DOI information for references in the revised manuscript.
Citation: https://doi.org/10.5194/egusphere-2022-1106-AC1
-
RC1: 'Comment on egusphere-2022-1106', Anonymous Referee #1, 23 Mar 2023
Major comments
This manuscript presents a framework for quantifying precipitation distributions at regionsl scales, which is interesting to the readers of the GMD and the community. If this is applied to CMIP5 and CMIP6, they found a few interesting results: (1) overall overestimation of light precipitation, (2) improvement in mid-latitude regions, and (3) degradation over the tripics. Overall, the authors did good job in listing details of the analysis method and result.
1. The intent of the authors seems quite clear. However, I do have impression that kwown/widely applied metrics are just applied over smaller regions - based on the CMIP6 regions. It is not clear to me what are the novelity of this manuscript.
2. Defining homogeneous region can also be subjective depending on the criteria chosen.
Overall, I do feel this well-done research and is worth to be published. I don't think one can have clear answer on #1 or #2, but do hope to see more clear scope and goals of the manuscript.
Citation: https://doi.org/10.5194/egusphere-2022-1106-RC1 -
AC2: 'Reply on RC1', Min-Seop Ahn, 20 May 2023
Major comments
This manuscript presents a framework for quantifying precipitation distributions at regionsl scales, which is interesting to the readers of the GMD and the community. If this is applied to CMIP5 and CMIP6, they found a few interesting results: (1) overall overestimation of light precipitation, (2) improvement in mid-latitude regions, and (3) degradation over the tripics. Overall, the authors did good job in listing details of the analysis method and result.
We thank the reviewer for the helpful and constructive comments. Please find our point-by-point response below.
1. The intent of the authors seems quite clear. However, I do have impression that kwown/widely applied metrics are just applied over smaller regions - based on the CMIP6 regions. It is not clear to me what are the novelity of this manuscript.
We think our manuscript has novelties in addition to the evaluation of simulated precipitation at regional scales. In this study we brought together a diverse suite of well-established precipitation distribution metrics and several new complementary metrics, providing a more comprehensive objective assessment of precipitation distributions than earlier studies that generally rely on only one or two metrics. This more comprehensive objective perspective has enabled us to assess which metrics are most informative, and which regions of the world have similar bias characteristics across CMIP models. In addition to that, via cluster analysis we propose a new suite of oceanic reference regions for evaluating simulated precipitation that are more homogeneous in precipitation character than those used in the IPCC AR6.
To clarify these points in the manuscript, we have revised the last paragraph of the introduction section. Below is the revised sentence:
“In this study, we propose a modified IPCC AR6 reference regions and a framework for regional scale quantification of simulated precipitation distributions, which is implemented into the PCMDI metrics package to enable researchers to readily use the metric collection in a common framework.”
2. Defining homogeneous region can also be subjective depending on the criteria chosen.
We agree with the reviewer that generally defining homogeneous regions is subjective depending on the criteria chosen. However, we did not use any criteria of precipitation for defining the homogeneous regions, rather, it is one of the outcomes of the K-means clustering. The K-means clustering algorithm defines the homogeneity regions by iterating to minimize the sum of distances between the cluster centroid and each cluster member. The only parameter for K-means clustering is the number of desired clusters. Here we use 3 for defining heavy, moderate, and light rain regions.
We have added more information about this in the revised manuscript. Below are the added/revised sentences in the method section:
“K-means clustering is an unsupervised machine learning algorithm that separates characteristics of a dataset into a given number of clusters without explicitly provided criteria. This method has been widely used because it is faster and simpler than other methods. Here, we use 3 clusters to define heavy, moderate, and light precipitation regions.”
Overall, I do feel this well-done research and is worth to be published. I don't think one can have clear answer on #1 or #2, but do hope to see more clear scope and goals of the manuscript.
We thank the reviewer’s comments that help improve our manuscript.
Citation: https://doi.org/10.5194/egusphere-2022-1106-AC2
-
AC2: 'Reply on RC1', Min-Seop Ahn, 20 May 2023
-
RC2: 'Comment on egusphere-2022-1106', Anonymous Referee #2, 12 May 2023
The paper provides a thorough model intercomparison as to how CMIP5 and 6 models simulate different characteristics of precipitation distributions across IPCC AR6 subregions and modified ocean regions. Across all metrics and globally, they note limited improvement in performance between CMIP5 and CMIP6, confined to some mid-latitude regions. Overall, the authors were thorough and clear in their justification and selection of metrics. I particularly like Figure 1 and the authors' methods to quantify observational discrepancy as well.
My main suggestions and comments are as follows:
1.) After reading the full manuscript, I am not exactly sure what the authors' 'Benchmarking Framework' is or how I could use it in my research, other than the consolidated metrics and possibly the methods for quantifying observational discrepancy. "Evaluation" and "Benchmarking" are used synonymously, despite distinct differences (see Abramowitz 2005, 2012). I am unsure what the benchmark is, or if the authors are establishing the benchmark for the next generation of CMIP. This could be clarified and stated explicitly within or following the second paragraph of the introduction and ideally in the abstract as well. It might also be beneficial to the reader to have the components of the framework summarized explicitly in the intro or discussion section.
2.) I recommend adding titles to each of the figures.
3.) Fig. 7 Caption '“Black, gray, blue, and red curves indicate the satellite-based observations, reanalysis, CMIP5 models, and CMIP6 modes, respectively.” I think is intended only for Fig. 6.
4.) In Table 3, is there not a citation for FracPR?
5.) The U.S. DOE Benchmarking Report is in the reference list, but I do not see it referenced in the text. I think instead you use Pendergrass, 2020?
Overall, I think these results should be published and will be beneficial to the research community, following revision to the text, primarily to address my first item of suggestion.
Citation: https://doi.org/10.5194/egusphere-2022-1106-RC2 -
AC3: 'Reply on RC2', Min-Seop Ahn, 20 May 2023
The paper provides a thorough model intercomparison as to how CMIP5 and 6 models simulate different characteristics of precipitation distributions across IPCC AR6 subregions and modified ocean regions. Across all metrics and globally, they note limited improvement in performance between CMIP5 and CMIP6, confined to some mid-latitude regions. Overall, the authors were thorough and clear in their justification and selection of metrics. I particularly like Figure 1 and the authors' methods to quantify observational discrepancy as well.
We thank the reviewer for the helpful and constructive comments. Please find our point-by-point response below.
My main suggestions and comments are as follows:
1.) After reading the full manuscript, I am not exactly sure what the authors' 'Benchmarking Framework' is or how I could use it in my research, other than the consolidated metrics and possibly the methods for quantifying observational discrepancy. "Evaluation" and "Benchmarking" are used synonymously, despite distinct differences (see Abramowitz 2005, 2012). I am unsure what the benchmark is, or if the authors are establishing the benchmark for the next generation of CMIP. This could be clarified and stated explicitly within or following the second paragraph of the introduction and ideally in the abstract as well. It might also be beneficial to the reader to have the components of the framework summarized explicitly in the intro or discussion section.
We appreciate the reviewer’s comment, which has helped us to clarify our benchmarking framework. As suggested, we have added more information about benchmarking in the second paragraph of the introduction section. Below are the added/revised sentences in the revised manuscript:
“As discussed in previous studies (e.g., Abramowitz 2012), our reference to model benchmarking implies model evaluation with community-established reference data sets, performance tests (metrics), variables, and spatial and temporal resolutions.”
“The current study provides a benchmarking framework focused on evaluating simulated precipitation distributions against multiple observations with well-established metrics and reference regions. To ensure consistent application of this framework, the metrics used herein are implemented and made available as part of the widely-used Program for Climate Model Diagnosis & Intercomparison (PCMDI) metrics package."
2.) I recommend adding titles to each of the figures.
We have added a title to Fig. 13, and all other figures have titles.
3.) Fig. 7 Caption '“Black, gray, blue, and red curves indicate the satellite-based observations, reanalysis, CMIP5 models, and CMIP6 modes, respectively.” I think is intended only for Fig. 6.
Thank you for pointing it out. We have replaced “curves” with “markers” in the caption.
4.) In Table 3, is there not a citation for FracPR?
We have added a reference for FracPRdays in the table.
5.) The U.S. DOE Benchmarking Report is in the reference list, but I do not see it referenced in the text. I think instead you use Pendergrass, 2020?
Thank you for pointing it out. We have cited both references in the revised manuscript.
Overall, I think these results should be published and will be beneficial to the research community, following revision to the text, primarily to address my first item of suggestion.
We are grateful for the comments and suggestions of the reviewers that helped improve the manuscript.
Citation: https://doi.org/10.5194/egusphere-2022-1106-AC3
-
AC3: 'Reply on RC2', Min-Seop Ahn, 20 May 2023
Min-Seop Ahn et al.
Min-Seop Ahn et al.
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 440 | 122 | 19 | 581 | 31 | 4 | 4 |
- HTML: 440
- PDF: 122
- XML: 19
- Total: 581
- Supplement: 31
- BibTeX: 4
- EndNote: 4
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1