Preprints
https://doi.org/10.5194/egusphere-2025-2311
https://doi.org/10.5194/egusphere-2025-2311
28 May 2025
 | 28 May 2025
Status: this preprint is open for discussion and under review for Earth System Dynamics (ESD).

Enhanced Climate Reproducibility Testing with False Discovery Rate Correction

Michael E. Kelleher and Salil Mahajan

Abstract. Simulating the Earth's climate is an important and complex problem, thus climate models are similarly complex, comprised of tens to hundreds of thousands of lines of code. In order to appropriately utilize the latest computational and software infrastructure advancements in Earth system models running on modern hybrid computing architectures to improve their performance, precision, accuracy, or all three; it is important to ensure that model simulations are repeatable and robust. This introduces the need for establishing statistical or non-bit-for-bit reproducibility, since bit-for-bit reproducibility may not always be achievable. Here, we propose a short-simulation ensemble-based test for an atmosphere model to evaluate the null hypothesis that modified model results are statistically equivalent to that of the original model. We implement this test in US Department of Energy's Energy Exascale Earth System Model (E3SM). The test evaluates a standard set of output variables across the two simulation ensembles and uses a false discovery rate correction to account for multiple testing. The false positive rates of the test are examined using re-sampling techniques on large simulation ensembles and are found to be lower than the currently implemented bootstrapping-based testing approach in E3SM. We also evaluate the statistical power of the test using perturbed simulation ensemble suites, each with a progressively larger magnitude of change to a tuning parameter. The new test is generally found to exhibit more statistical power than the current approach, being able to detect smaller changes in parameter values with higher confidence.

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.
Share
Michael E. Kelleher and Salil Mahajan

Status: open (until 11 Jul 2025)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse
Michael E. Kelleher and Salil Mahajan
Michael E. Kelleher and Salil Mahajan

Viewed

Total article views: 89 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
73 13 3 89 5 2
  • HTML: 73
  • PDF: 13
  • XML: 3
  • Total: 89
  • BibTeX: 5
  • EndNote: 2
Views and downloads (calculated since 28 May 2025)
Cumulative views and downloads (calculated since 28 May 2025)

Viewed (geographical distribution)

Total article views: 90 (including HTML, PDF, and XML) Thereof 90 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Latest update: 23 Jun 2025
Download
Short summary
Building numerical models of the Earth is a complex task that scientists and engineers around the world work on. It's important to be able to replicate results accurately to help advance science. This study uses a statistical method to reduce false positive errors when comparing two sets of simulations to see if they agree with each other. This approach helps identify if changes made to the model's code result in unexpected effects.
Share