the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 22 Oct 2024
Implementation and validation of a supermodelling framework into CESM version 2.1.5
Abstract. Here we present a framework for the first atmosphere-connected supermodel using state-of-the-art atmospheric models. The Community Atmosphere Model (CAM) versions 5 and 6 exchange information interactively while running, a process known as supermodeling. The primary goal of this approach is to synchronize the models, allowing them to compensate for each other's systematic errors in real-time, in part by increasing the dimentionality of the system.
In this study, we examine a single untrained supermodel where each model version is equally weighted in creating pseudo-observations. We demonstrate that the models synchronize well without decreased variability, particularly in storm track regions, across multiple timescales and for variables where no information has been exchanged. Synchronization is less pronounced in the tropics, and in regions of lesser synchronization we observe a decrease in high-frequency variability. Additionally, the low-frequency modes of variability (North Atlantic Oscillation and Pacific North American Pattern) are not degraded compared to the base models. For some variables, the mean bias is reduced compared to control simulations of each model version as well as the non-interactive ensemble mean.
-
Notice on discussion status
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
-
Preprint
(15872 KB)
-
Supplement
(9291 KB)
-
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(15872 KB) - Metadata XML
-
Supplement
(9291 KB) - BibTeX
- EndNote
- Final revised paper
Journal article(s) based on this preprint
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-2682', Anonymous Referee #1, 22 Nov 2024
This paper describes a supermodelling framework that combines the effects of the physics parameterization suites from two versions of the Community Atmosphere Model. The physics suites are combined by nudging each version (model component) to the averaged state on a periodic time interval, as in data assimilation.
Supermodelling is an interesting idea with interesting potential applications, but I think that the manuscript is not ready for publication for three reasons. First, the method is not described in enough detail to allow a reader to reproduce the results. For instance, I couldn’t find a tag of the code listed in the manuscript that would allow me to recreate the figures in the manuscript. Furthermore, the scripts have hardwired paths (not variables) with little instruction given as to what lines in the scripts need to be modified. Second, the main application cited is improvement of supermodeled climatologies over the individual components, but only one or two improved fields are shown in the manuscript. The authors argue that improvement will come with tuning, but one might expect to see improvement even with simple equal weighting of the components. It might interest readers to see more discussion of which fields are improved and which are degraded, along with any insights that the authors have regarding the reasons that some fields are improved and others are not. Third, there are many typos in the manuscript, some of which impede understanding of the meaning. I list just a sample of them below. My recommendation is that the authors invest more time in the manuscript and deliver a more polished version.
Minor comments:
This part of the caption of Fig. 2 seems to have typos: “SUMO5 (teal); supermodel which uses CAM5-physics (SUMO5); supermodel supermodel which uses CAM6-physics (SUMO6) (purple) at location”
Lines 32–33: “Additionally, biases which are shared across the individual models in an NIE cannot be corrected due to the linear nature of post-process averaging.” But does the supermodel succeed in correcting such shared biases? Can the authors offer any insight into when and why shared biases can be corrected?
Line 80: “the adaptation of the existing nudging toolbox (reference)”. Typo. Please include whatever reference you’re referring to here.
Lines 178–180: “We show results for four experiments: CAM5, CAM6, the supermodel which uses CAM5-physics, but is nudged to the combined state, SUMO5, and the supermodel which uses CAM6-physics, SUMO6.” Does SUMO6 nudge to the combined state? If not, how is it different from CAM6? The sentence ends before the SUMO6 configuration is clearly described.
Line 195: “(Fig. ??, right column)”. Typo.
Lines 195–196: “Correlations between the SUMO5 and SUMO6 experiments are much higher (3, left column)” How can a correlation be as high as 3? Correlations must lie between -1 and 1.
Fig. 4: I’m surprised that CAM5 and CAM6 have such similar distributions of wind speed. Is there a computational or physical constraint on this distribution?
Line 258: “One reason to develop supermodels lies in their potential to have smaller mean-field biases.” In the example of surface precipitation shown in Fig. 7, the spatial pattern of errors looks similar (shared) in CAM5 and CAM6. The hope implied in lines 32–33 was that such error could be reduced below both components by supermodelling. But Fig. 7 shows that this doesn’t happen for surface precipitation. Do the authors have an explanation for this failure?
Line 258: “The work presented her” Typo.
Lines 278–279: “Positive values (grey) indicate that the SUPERense is outperforming the NIEnse and vise-versa.” Are positive values grey or green?
Lines 289–290: “This circumvents the in CESM costly initialization stage and introduces effectively PAUSE/RESUME capability (Fig. 1).” Typos?
Lines 300-301: “To test our implementation we linked the CAM5/CAM6 atmosphere and confirmed that synchronization across various temporal scales and variables Even though the supermodels only exchange limited information . . .” What did you confirm?
Lines 317–319: “To create all model runs and build your own supermodel, refer to Chapman et al. (2024). This second repository contains the setup for the SuperModel and its constituent models, including source modifications, model build scripts, and namelists for running the described CAM versions.” Is a tag of the CESM repository listed somewhere so that readers can reproduce the figures in the paper identically?
Citation: https://doi.org/10.5194/egusphere-2024-2682-RC1 -
CC1: 'Comment on egusphere-2024-2682', Gregory Duane, 29 Nov 2024
The benefits of combining models in an untrained supemodel are slightly greater than indicated by the authors on line 260 in the statement “any improvement in model bias is fortuitous”. It is well known that the state-of-the-art (pre-supermodeling) approach of combining the outputs of multiple climate models in a simple average almost always results in improvement (Reicher and Kim 2008)1 . The only explanation I know of is that the model error of each model can be regarded as random error in, say, some large parameter space. The random errors of the different models then tend to compensate, on average. That benefit should be retained in a supermodel, and probably enhanced, because the model errors can compensate before they propagate dynamically to other scales and other parts of the model. Of course, with only two models, the expected improvement from this effect is expected to be small, reducing RMS error by a factor of only \sqrt{2} for output averaging if the errors are truly random, and even less if, as here, they are not. But the improvement might be somewhat greater for supermodeling.
It would be interesting to know if there is any evidence at all of improvement in the supermodel that would not have come from an output average. For reasons discussed by Duane and Shen (2023)2, such improvements are often manifest in representations of localized structures, rather than in reductions in RMS error. If there is any evidence of such localized improvements which appear robust, even if they are small, it should at least be mentioned in the text. (Illustrations and any additional details could of course be included in the Supplemental Material.)
1. Reichler, T., and J. Kim, 2008: How well do coupled models simulate today’s climate? Amer. Meteor. Soc., 89, 303–311, https://doi.org/10.1175/BAMS-89-3-303
2 Duane, G.S. and Shen, M.-L., 2023: Synchronization of alternative models in a supermodel and the learning of critical behavior. J. Atmos. Sci., 80, 1565-1584, https://doi.org/10.1175/JAS-D-22-0113.1
Citation: https://doi.org/10.5194/egusphere-2024-2682-CC1 -
AC3: 'Reply on CC1', Will Chapman, 11 Mar 2025
Hi Greg,
We have incorporated your feedback into the new manuscript and this manifests in a few areas of the manuscript. We post a few lines below to show this, but it echos in other parts of the revised document. For example, we have significantly altered the line you quote in the paper as well as citing the manuscripts to strengthen that evidence.
"One motivation for developing supermodels is their potential to reduce mean-field biases. In our work, we emphasize the supermodeling implementation connecting CESM components without performing any training—any bias improvements are muted. Therefore we do not anticipate substantial reduction in bias, but there may be minor improvements beyond that from averaging of non-interactive simulations because error compensation at an early stage \citep{schevenhoven2023supermodeling, duane2023synchronization}. "
"Taking advantage of model diversity compensates for individual model bias errors and allows more complex dynamical behavior. However, this is often evidenced in representations of localized structures, rather than in reductions in mean squared error \citep{duane2023synchronization}. "Citation: https://doi.org/10.5194/egusphere-2024-2682-AC3
-
AC3: 'Reply on CC1', Will Chapman, 11 Mar 2025
-
RC2: 'Comment on egusphere-2024-2682', Anonymous Referee #2, 13 Jan 2025
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2024/egusphere-2024-2682/egusphere-2024-2682-RC2-supplement.pdf
-
AC1: 'Comment on egusphere-2024-2682', Will Chapman, 11 Mar 2025
We thank the reviewers for their thoughtful comments, and have included an acknowledgment to their work in the manuscript. We highlight that the manuscript has gone through a few significant changes due to this review and we believe it is in a much stronger condition. It has been refined greatly with an eye towards readability and flow. We include many more error statistics (see table 1 in the supplemental material). The github repository has been restructured for user clarity (which was a concern of reviewer 1, and this has resulted in new versions on zenodo), and we highlight the improved use-ability of the code and modeling framework. Additionally, the GPCP dataset has been remapped with a new interpolation and this updated many figures, though the broad findings are unchanged.
We have included our full response in the attached document which contains point-by-point responses and revisions to the final manuscript.
-
AC2: 'Comment on egusphere-2024-2682 - response to RC2', Will Chapman, 11 Mar 2025
We thank the reviewers for their thoughtful comments, and have included an acknowledgment to their work in the manuscript. We highlight that the manuscript has gone through a few significant changes due to this review and we believe it is in a much stronger condition. It has been refined greatly with an eye towards readability and flow. We include many more error statistics (see table 1 in the supplemental material). The github repository has been restructured for user clarity (which was a concern of reviewer 1, and this has resulted in new versions on zenodo), and we highlight the improved use-ability of the code and modeling framework. Additionally, the GPCP dataset has been remapped with a new interpolation and this updated many figures, though the broad findings are unchanged.
We have included our full response in the attached document which contains point-by-point responses and revisions to the final manuscript.
Interactive discussion
Status: closed
-
RC1: 'Comment on egusphere-2024-2682', Anonymous Referee #1, 22 Nov 2024
This paper describes a supermodelling framework that combines the effects of the physics parameterization suites from two versions of the Community Atmosphere Model. The physics suites are combined by nudging each version (model component) to the averaged state on a periodic time interval, as in data assimilation.
Supermodelling is an interesting idea with interesting potential applications, but I think that the manuscript is not ready for publication for three reasons. First, the method is not described in enough detail to allow a reader to reproduce the results. For instance, I couldn’t find a tag of the code listed in the manuscript that would allow me to recreate the figures in the manuscript. Furthermore, the scripts have hardwired paths (not variables) with little instruction given as to what lines in the scripts need to be modified. Second, the main application cited is improvement of supermodeled climatologies over the individual components, but only one or two improved fields are shown in the manuscript. The authors argue that improvement will come with tuning, but one might expect to see improvement even with simple equal weighting of the components. It might interest readers to see more discussion of which fields are improved and which are degraded, along with any insights that the authors have regarding the reasons that some fields are improved and others are not. Third, there are many typos in the manuscript, some of which impede understanding of the meaning. I list just a sample of them below. My recommendation is that the authors invest more time in the manuscript and deliver a more polished version.
Minor comments:
This part of the caption of Fig. 2 seems to have typos: “SUMO5 (teal); supermodel which uses CAM5-physics (SUMO5); supermodel supermodel which uses CAM6-physics (SUMO6) (purple) at location”
Lines 32–33: “Additionally, biases which are shared across the individual models in an NIE cannot be corrected due to the linear nature of post-process averaging.” But does the supermodel succeed in correcting such shared biases? Can the authors offer any insight into when and why shared biases can be corrected?
Line 80: “the adaptation of the existing nudging toolbox (reference)”. Typo. Please include whatever reference you’re referring to here.
Lines 178–180: “We show results for four experiments: CAM5, CAM6, the supermodel which uses CAM5-physics, but is nudged to the combined state, SUMO5, and the supermodel which uses CAM6-physics, SUMO6.” Does SUMO6 nudge to the combined state? If not, how is it different from CAM6? The sentence ends before the SUMO6 configuration is clearly described.
Line 195: “(Fig. ??, right column)”. Typo.
Lines 195–196: “Correlations between the SUMO5 and SUMO6 experiments are much higher (3, left column)” How can a correlation be as high as 3? Correlations must lie between -1 and 1.
Fig. 4: I’m surprised that CAM5 and CAM6 have such similar distributions of wind speed. Is there a computational or physical constraint on this distribution?
Line 258: “One reason to develop supermodels lies in their potential to have smaller mean-field biases.” In the example of surface precipitation shown in Fig. 7, the spatial pattern of errors looks similar (shared) in CAM5 and CAM6. The hope implied in lines 32–33 was that such error could be reduced below both components by supermodelling. But Fig. 7 shows that this doesn’t happen for surface precipitation. Do the authors have an explanation for this failure?
Line 258: “The work presented her” Typo.
Lines 278–279: “Positive values (grey) indicate that the SUPERense is outperforming the NIEnse and vise-versa.” Are positive values grey or green?
Lines 289–290: “This circumvents the in CESM costly initialization stage and introduces effectively PAUSE/RESUME capability (Fig. 1).” Typos?
Lines 300-301: “To test our implementation we linked the CAM5/CAM6 atmosphere and confirmed that synchronization across various temporal scales and variables Even though the supermodels only exchange limited information . . .” What did you confirm?
Lines 317–319: “To create all model runs and build your own supermodel, refer to Chapman et al. (2024). This second repository contains the setup for the SuperModel and its constituent models, including source modifications, model build scripts, and namelists for running the described CAM versions.” Is a tag of the CESM repository listed somewhere so that readers can reproduce the figures in the paper identically?
Citation: https://doi.org/10.5194/egusphere-2024-2682-RC1 -
CC1: 'Comment on egusphere-2024-2682', Gregory Duane, 29 Nov 2024
The benefits of combining models in an untrained supemodel are slightly greater than indicated by the authors on line 260 in the statement “any improvement in model bias is fortuitous”. It is well known that the state-of-the-art (pre-supermodeling) approach of combining the outputs of multiple climate models in a simple average almost always results in improvement (Reicher and Kim 2008)1 . The only explanation I know of is that the model error of each model can be regarded as random error in, say, some large parameter space. The random errors of the different models then tend to compensate, on average. That benefit should be retained in a supermodel, and probably enhanced, because the model errors can compensate before they propagate dynamically to other scales and other parts of the model. Of course, with only two models, the expected improvement from this effect is expected to be small, reducing RMS error by a factor of only \sqrt{2} for output averaging if the errors are truly random, and even less if, as here, they are not. But the improvement might be somewhat greater for supermodeling.
It would be interesting to know if there is any evidence at all of improvement in the supermodel that would not have come from an output average. For reasons discussed by Duane and Shen (2023)2, such improvements are often manifest in representations of localized structures, rather than in reductions in RMS error. If there is any evidence of such localized improvements which appear robust, even if they are small, it should at least be mentioned in the text. (Illustrations and any additional details could of course be included in the Supplemental Material.)
1. Reichler, T., and J. Kim, 2008: How well do coupled models simulate today’s climate? Amer. Meteor. Soc., 89, 303–311, https://doi.org/10.1175/BAMS-89-3-303
2 Duane, G.S. and Shen, M.-L., 2023: Synchronization of alternative models in a supermodel and the learning of critical behavior. J. Atmos. Sci., 80, 1565-1584, https://doi.org/10.1175/JAS-D-22-0113.1
Citation: https://doi.org/10.5194/egusphere-2024-2682-CC1 -
AC3: 'Reply on CC1', Will Chapman, 11 Mar 2025
Hi Greg,
We have incorporated your feedback into the new manuscript and this manifests in a few areas of the manuscript. We post a few lines below to show this, but it echos in other parts of the revised document. For example, we have significantly altered the line you quote in the paper as well as citing the manuscripts to strengthen that evidence.
"One motivation for developing supermodels is their potential to reduce mean-field biases. In our work, we emphasize the supermodeling implementation connecting CESM components without performing any training—any bias improvements are muted. Therefore we do not anticipate substantial reduction in bias, but there may be minor improvements beyond that from averaging of non-interactive simulations because error compensation at an early stage \citep{schevenhoven2023supermodeling, duane2023synchronization}. "
"Taking advantage of model diversity compensates for individual model bias errors and allows more complex dynamical behavior. However, this is often evidenced in representations of localized structures, rather than in reductions in mean squared error \citep{duane2023synchronization}. "Citation: https://doi.org/10.5194/egusphere-2024-2682-AC3
-
AC3: 'Reply on CC1', Will Chapman, 11 Mar 2025
-
RC2: 'Comment on egusphere-2024-2682', Anonymous Referee #2, 13 Jan 2025
The comment was uploaded in the form of a supplement: https://egusphere.copernicus.org/preprints/2024/egusphere-2024-2682/egusphere-2024-2682-RC2-supplement.pdf
-
AC1: 'Comment on egusphere-2024-2682', Will Chapman, 11 Mar 2025
We thank the reviewers for their thoughtful comments, and have included an acknowledgment to their work in the manuscript. We highlight that the manuscript has gone through a few significant changes due to this review and we believe it is in a much stronger condition. It has been refined greatly with an eye towards readability and flow. We include many more error statistics (see table 1 in the supplemental material). The github repository has been restructured for user clarity (which was a concern of reviewer 1, and this has resulted in new versions on zenodo), and we highlight the improved use-ability of the code and modeling framework. Additionally, the GPCP dataset has been remapped with a new interpolation and this updated many figures, though the broad findings are unchanged.
We have included our full response in the attached document which contains point-by-point responses and revisions to the final manuscript.
-
AC2: 'Comment on egusphere-2024-2682 - response to RC2', Will Chapman, 11 Mar 2025
We thank the reviewers for their thoughtful comments, and have included an acknowledgment to their work in the manuscript. We highlight that the manuscript has gone through a few significant changes due to this review and we believe it is in a much stronger condition. It has been refined greatly with an eye towards readability and flow. We include many more error statistics (see table 1 in the supplemental material). The github repository has been restructured for user clarity (which was a concern of reviewer 1, and this has resulted in new versions on zenodo), and we highlight the improved use-ability of the code and modeling framework. Additionally, the GPCP dataset has been remapped with a new interpolation and this updated many figures, though the broad findings are unchanged.
We have included our full response in the attached document which contains point-by-point responses and revisions to the final manuscript.
Peer review completion
Journal article(s) based on this preprint
Viewed
| HTML | XML | Total | Supplement | BibTeX | EndNote | |
|---|---|---|---|---|---|---|
| 438 | 135 | 39 | 612 | 67 | 35 | 57 |
- HTML: 438
- PDF: 135
- XML: 39
- Total: 612
- Supplement: 67
- BibTeX: 35
- EndNote: 57
Viewed (geographical distribution)
| Country | # | Views | % |
|---|
| Total: | 0 |
| HTML: | 0 |
| PDF: | 0 |
| XML: | 0 |
- 1
Francine Schevenhoven
Judith Berner
Noel Keenlyside
Ingo Bethke
Ping-Gin Chiu
Alok Gupta
Jesse Nusbaumer
The requested preprint has a corresponding peer-reviewed final revised paper. You are encouraged to refer to the final revised version.
- Preprint
(15872 KB) - Metadata XML
-
Supplement
(9291 KB) - BibTeX
- EndNote
- Final revised paper