| 11 May 2026
A robustness diagnostic framework for NMIP ensembles: Application to NMIP2 soil N2O emission estimates
Abstract. Multi-model ensembles, such as the global N2O Model Intercomparison Project (NMIP), are essential for quantifying terrestrial nitrous oxide (N2O) fluxes; however, interpreting where ensemble results are reliable and where they are not remains challenging. Here, we propose a robustness diagnostic framework (RDF) that classifies each grid cell into four categories: Robust-increase, Robust-decrease, Divergent and Uncertain, based on the three metrics of ensemble-mean change, model agreement on the sign of change and inter-model standard deviation. By explicitly separating directional disagreement (Divergent) from quantitative disagreement (Uncertain), the RDF reveals the nature of model uncertainty rather than its magnitude alone. We applied the RDF to soil N2O emission estimates from eight terrestrial biosphere models participating in NMIP phase 2 (NMIP2), comparing pre-industrial (1850s) and contemporary (2010s) periods. Of 56,852 valid grid cells, 40.8 % were classified as Robust-increase, 2.6 % as Robust-decrease, 36.1 % as Divergent and 20.5 % as Uncertain. Stratification by land-use type revealed that the nature of uncertainty differed fundamentally: cropland-dominated regions were dominated by Uncertain (72.6 %), indicating an agreement on the direction of N2O increase but a large quantitative spread, whereas forest-dominated regions were dominated by Divergent (50.0 %), indicating disagreement on the direction of change itself. Pasture-dominated regions exhibited the highest robustness (59.1 % Robust-increase). The inter-model spread correlated strongly with nitrogen input intensity (Spearman ρ = 0.75), and the Divergent to Uncertain transition followed a gradient of cropland fraction. These contrasting patterns implied that different model improvement strategies were needed: observational benchmarking of emission magnitudes for croplands, and improved process understanding of the competition between carbon dioxide (CO2) fertilization and warming effects in forests. The proposed framework is general and applicable to any multi-model ensemble of biogeochemical change.
Review comments for A robustness diagnostic framework for NMIP ensembles: Application to NMIP2 soil N2O emission estimates (egusphere-2026-1817)
This study proposes a robustness diagnostic framework (RDF) to classify grid-cell-level changes in NMIP2 soil N₂O emissions into four categories: Robust-increase, Robust-decrease, Divergent, and Uncertain. The topic is relevant to multi-model ensemble assessment, and the application to NMIP2 is potentially useful. However, I doubt whether the manuscript provides a sufficiently substantial methodological advance for publication in Geoscientific Model Development. In addition, several key conclusions regarding land-use-specific uncertainty and its drivers are strongly influenced by the current classification design and by the use of grid-cell-level mixed land-use fractions. I have the following concerns and suggestions for the author to consider.
1. The central methodological contribution of the manuscript is the RDF classification scheme, which combines three metrics: ensemble-mean change, model agreement on the sign of change, and inter-model standard deviation. These are all standard diagnostics in multi-model ensemble analysis. The main novelty appears to be the grouping of these diagnostics into four qualitative classes. Although this is useful for interpretation, I am not convinced that it constitutes a sufficiently substantial methodological advance for GMD in its present form. The manuscript should more clearly demonstrate what is new relative to existing robustness and model-agreement approaches, especially those already used in climate-model ensemble studies. A more convincing contribution would require a systematic comparison between the proposed RDF and existing diagnostics, such as sign agreement maps or robustness maps. The current manuscript reads more as an application of existing ensemble diagnostics to NMIP2 outputs than as a new model assessment method.
2. The definition of “Uncertain” mechanically classifies high-N-input regions as uncertain. The definition of the Uncertain class is based on sign agreement combined with an absolute inter-model standard deviation threshold. This design makes grid cells with high N input rates much more likely to be classified as Uncertain, because high N inputs naturally lead to larger absolute N₂O emissions and therefore larger absolute inter-model spread. As a result, almost all major agricultural regions, including eastern China, India, western Europe, the US Corn Belt, and southeastern Brazil, were classified as Uncertain (Figure 2a).
The manuscript’s Figure 3 shows that model agreement is very high for cropland-dominated grid cells, and the median KA for croplands reached 1.000. The classification as Uncertain therefore does not primarily reflect disagreement in the direction of N₂O change, but rather the use of an absolute SD threshold in regions where the value itself is large. This raises a fundamental question: does the Uncertain category identify genuinely low-confidence regions, or does it primarily identify high-emission/high-N-input regions with large absolute fluxes? The framework would be more robust if it considered relative uncertainty, signal-to-noise ratio, normalized SD, or uncertainty relative to the magnitude of the ensemble-mean change.
3. The analysis of cropland-, forest-, and pasture-dominated regions is one of the central components of the manuscript. However, the current classification is based on grid-cell-level land-use fractions, with thresholds of cropland ≥ 0.3, forest ≥ 0.5, and pasture ≥ 0.3. However, most 0.5° grid cells are mixtures of different land use types. Under the current definition, a grid cell with cropland fraction slightly above 0.3 can be classified as cropland/pasture dominated even if another biome or land-use type occupies a larger fraction of the same grid cell. The terms “cropland-dominated” and “pasture-dominated” may be misleading. A more appropriate analysis would be to conduct the diagnostic at the biome or PFT level, if such outputs are available from NMIP2 models, or at least to use stricter land-use classification criteria.
4. The discussion attributes cropland uncertainty mainly to differences in N-input sensitivity and forest divergence to the competing effects of CO₂ fertilization and warming. These interpretations are plausible, but they remain largely speculative in the current manuscript. NMIP2 includes a series of factorial experiments designed to isolate the effects of different drivers. These experiments should be used to quantify the contributions of different drivers to inter-model disagreement. This would substantially strengthen the manuscript and move it from a descriptive classification exercise toward a mechanistic model-diagnostic study. Without using these factorial simulations, the manuscript cannot robustly identify the drivers of uncertainty. The current SH1-only analysis can describe where models disagree, but it cannot adequately explain why they disagree.
Minor comments:
Line 28: “Forster” rather than “Foster”
Line 74-75: “mineral nitrogen fertilization and manure nitrogen common driving data” The sentence is grammatically incorrect.
Lines 154–155: The manuscript states that land-use categories are not mutually exclusive. However, Figure 2b is described as showing “dominant land-use type.” The authors should clarify how overlapping grid cells are handled in the map and in the statistics.
Figure3: The individual panels are small, and some labels, legends, and axis text are difficult to read.
Line 350: “because we adapted data-driven thresholds” should be “because we adopted data-driven thresholds.”
Line 362: “thresholds ... adapted here” should be “thresholds ... adopted here.”
Line 397: “The NMIPs soil N₂O emission data” should be corrected to “The NMIP2 soil N₂O emission data”.