Directive-based GPU acceleration of anthropogenic and biosphere flux computations in ICON-ART (v2026.04)
Abstract. Atmospheric composition models require efficient computation of surface fluxes from anthropogenic emissions and terrestrial biosphere exchange. This is particularly true for applications such as ensemble-based inverse emission modeling where these calculations constitute significant performance bottlenecks. We present the directive-based GPU acceleration and comprehensive refactoring of two critical surface-flux modules within the ICON-ART atmospheric modeling framework: the Online Emission Module (OEM) for anthropogenic emissions and the Vegetation Photosynthesis and Respiration Model (VPRM) for biospheric Carbon Dioxide (CO2) exchange. The implementation employs OPENACC compiler directives to expose inherent data parallelism while maintaining a single portable source code for both CPU and GPU architectures.
The optimization strategy addresses fundamental incompatibilities between the original CPU-oriented algorithms and GPU execution requirements through three principal modifications: (i) elimination of runtime string comparisons via pre-computed integer-indexed lookup tables constructed during initialization, (ii) temporal decomposition of computations into sections with distinct update frequencies (initialization (once), boundary conditions (periodic), temporal scaling (hourly), and emission kernels (every timestep)) to minimize redundant calculations, and (iii) reorganization of loop hierarchies to maximize memory coalescing across ICON’s horizontal grid dimension. The hybrid CPU–GPU architecture executes initialization and I/O-intensive boundary updates on the host while offloading the computationally dominant temporal scaling and emission kernels to the device, with all tracer tendencies accumulated in device-resident arrays to eliminate host–device transfers within the physics integration loop.
Scientific equivalence between CPU and GPU implementations is validated using a probabilistic testing framework, which establishes tolerance envelopes from perturbed CPU ensemble runs. This is consistent with spatial comparisons of simulated CO2 concentration fields over a European domain, which demonstrate pixel-wise agreement within ±0.01 %, corresponding to the numerical noise floor from floating-point arithmetic variations. Performance benchmarks on NVIDIA Grace–Hopper (GH200) superchips of the ALPS supercomputer in Switzerland reveal speedups of ∼1.9× on a single GPU node to 5.8× on five GPU nodes relative to a 278-rank CPU baseline. Despite 54–93 % higher absolute energy consumption on GPUs, the energy-delay product improves by 20–67 %, demonstrating favorable computational efficiency for time-critical applications.
These advances enable ensemble-based atmospheric inversions with hundreds of tracers at kilometer-scale resolution within operationally feasible time constraints, supporting assimilation of dense satellite observations from instruments such as TROPOMI and the forthcoming CO2M mission. The GPU-enabled modules have been successfully deployed in recent ICON-ART applications for regional greenhouse-gas inversion and ensemble data assimilation. The single-source, directive-based implementation ensures long-term maintainability and portability across GPU generations while providing a reusable template for accelerating additional model components.
## Summary
The manuscript describes the GPU port of two surface-flux modules in ICON-ART: the Online Emission Module (OEM), which computes anthropogenic emissions at runtime from sectoral inventories and temporal and vertical profiles, and the Vegetation Photosynthesis and Respiration Model (VPRM), which computes biospheric CO₂ fluxes from satellite vegetation indices and meteorological drivers. Both modules can generate ensembles of perturbed fluxes, one tracer per ensemble member, for ensemble-based inverse modelling of greenhouse-gas emissions. With the few hundred ensemble tracers such inversions ideally use, at kilometre-scale resolution, the authors argue that these modules become a significant part of the model cost. Since the rest of ICON already runs on GPUs, porting them is a natural next step.
The port uses OpenACC directives, following the approach of the existing ICON GPU port, with the stated primary aim of keeping a single source that compiles and runs correctly on both CPU and GPU. The authors identify the original CPU-oriented code structure as poorly suited to GPU execution and refactor it in four main ways:
- **Lookup tables instead of string comparisons:** string-based searches for tracer, category and ensemble indices are replaced with integer lookup tables and precomputed emission weights, built once at initialisation.
- **Separation by update frequency:** the computation is split into initialisation (once), boundary-condition updates (periodic), temporal scaling (hourly) and the emission kernels (every timestep). The first two stay on the host, while the hourly and per-timestep parts run on the device, and tendencies are accumulated in device-resident arrays to avoid host–device transfers inside the time loop.
- **Loop reordering:** loop nests are reorganised so that the horizontal (`jc`) and vertical dimensions are collapsed into the parallel iteration space, for coalesced memory access on the GPU.
- **Ensemble handling:** dedicated kernels and lookup tables for ensemble tracers, including vegetation-class-specific scaling of GPP and respiration in VPRM.
The implementation is presented in detail as pseudocode and Fortran/OpenACC listings. Performance is evaluated on ALPS (NVIDIA GH200) for a 3.3 km European CO₂ configuration (∼306,000 cells, 60 levels, 45 ensemble members), comparing whole-model wall-clock times on one to five GPU nodes with a single-node CPU baseline. The authors report speedups of 1.9× on one GPU node and 5.8× on five, near-ideal strong scaling of the two modules, and an improved energy–delay product despite higher energy consumption on GPU. CPU–GPU consistency is assessed with the probtest framework and with maps of relative differences in simulated CO₂, which the authors find to agree within ±0.01%. The ported modules have already been used in recent ICON-ART inversion studies.
## General assessment
The paper is well written, fits GMD's scope and will be useful to the ICON-ART community. The single-source OpenACC approach, the archived input data and reproduction scripts, the use of probtest, and the attempt to report energy alongside time to solution are all welcome.
I particularly appreciated the detailed discussion of loop ordering and memory access patterns in Sects. 3–4. Papers on GPU ports rarely explain why the original loop structure performs poorly on the device and how the data layout and iteration space were reorganised to fix it. This makes the paper useful as a practical guide for porting other ICON-ART components, beyond the two modules treated here. Some of this material could be presented more concisely, and the memory layout itself could be made more explicit (see the minor comments), but I would encourage the authors to keep this discussion at its core.
My main concerns are with how performance is reported. The headline speedups are not like-for-like comparisons, and the effect of the refactor itself is never measured. This matters because the paper's stated primary goal is a single portable source for CPU and GPU (L8, L91–92), but the changes made for the GPU may well have slowed down the CPU code path, and this is neither acknowledged nor quantified. Elsewhere, the implementation sections could be shortened, mainly by removing the overlap between pseudocode and listings and some repetition in Sects. 6 and 8.
## Major comments
### 1. Performance claims: the speedups are not like-for-like, and the effect of the refactor is not measured
**(a) The headline speedups are misleading.** The abstract (L21–23) and conclusions (L1174–1177) report a speedup of "∼1.9× on a single GPU node to 5.8× on five GPU nodes relative to a 278-rank CPU baseline". Readers will take 5.8× to mean "5.8× faster than a comparable CPU run". In fact it compares five GPU nodes with *one* CPU node, mixing strong scaling with the architecture comparison. The authors acknowledge this (L949–950) but still lead with 5.8× in the abstract, Fig. 2 and the conclusions. Please report only like-for-like comparisons there (equal node counts, or CPU strong scaling over the same 1–5 nodes).
**(b) The effect of the refactor on either architecture is not measured, which leaves the portability claim unsupported.** The paper's primary design goal is "a single source that compiles and runs correctly on both CPU and GPU" (L91–92). All three optimisations in Sect. 4.1 change the code that also runs on CPU, but their effect on CPU performance is never measured. The authors state that no regression was "observed" and that the effects were "not isolated" (L582–585), with no numbers.
This is a real concern, not a formality. L579–582 argue that the new loop order improves CPU cache locality because "consecutive iterations of the inner loop access contiguous memory". Under the loop order of Eq. (4) (`jb -> jc -> k -> nt -> nc`), however, the innermost CPU loops run over `nt` and `nc`. In `total_weight(jc,k,nt,nc,jb)` these have strides of `nproma·Nlev` and `nproma-Nlev-Ntracer`, not 1. Moving `jc` out of the innermost position is the opposite of the usual CPU layout, where the contiguous nproma dimension is innermost for SIMD vectorisation. The refactor may therefore have slowed down the CPU code. Since the CPU baseline already includes the refactor (L584), this would also inflate the reported CPU-to-GPU ratios.
The authors do not need to claim performance portability, but the paper should say whether it is achieved. Please provide:
- CPU timings for OEM+VPRM before and after the refactor (same build and configuration), ideally separating the effect of the loop reordering from that of removing the string lookups;
- a GPU comparison that shows which of the three optimisations matters (e.g. against a direct port of the original loop structure);
- a sensitivity check of the CPU baseline (nproma and/or rank count). The 278 ranks, no OpenMP and nproma = 8 (L931–933) are justified only as ICON defaults, not as tuned for Grace;
**(c) The whole-model figures credit this work with the ICON GPU port.** The 1.9×/5.8× figures are whole-model times, and most of that gain comes from the existing ICON port (Lapillonne et al., 2026). For the modules this paper is about, OEM+VPRM goes from 350 s to 289 s on one node (1.21×, L957–958), and their share of runtime *rises* from 12.7% to 20.1% (Table 6). Relative to the rest of the model, the ported modules are less efficient on GPU than on CPU, which contradicts L962–963 ("prevents these modules from becoming a bottleneck"). The module-level result should be the primary claim, with a breakdown of the 289 s by section (initialisation, boundary update, hourly scaling, per-timestep kernels, host–device transfers).
**(d) The strong-scaling statements need correcting.** Fig. 2(b) plots speedup relative to the single CPU node, so both curves combine the architecture gain with multi-node scaling. Parallel speedup should be computed from the GPU run times themselves, relative to one GPU node, with the CPU comparison shown separately. On that basis the five-node speedup is 1438/475 ≈ 3.0× for the whole model and 289/59 ≈ 4.9× for OEM+VPRM. The paper's 60.5% whole-model efficiency is consistent with this, because the CPU baseline cancels out. The module-level efficiency (about 98%) is never stated, however.
Instead, L983–985 call 350/59 = 5.93× a "theoretical ideal" that the result slightly exceeds. That ratio *is* the measured speedup, so the argument is circular. Measured against the CPU baseline, the ideal at five nodes would be about 1.21 × 5 ≈ 6.05×, which the result does not exceed. The caption of Fig. 2 contradicts itself here: it describes 5.93× as "relative to the single node GPU baseline", while its own check (289 s / 5 = 57.8 s expected vs. 59 s measured) corresponds to 4.9×. The module scaling is a good result, but it is not superlinear, and L1030 repeats the same mix-up. L1030–1032 also draw conclusions about larger ensembles from strong-scaling data, although larger ensembles were not tested.
### 2. Pseudocode and code listings
The implementation is described twice: in pseudocode (Algorithms 1–3) and in Fortran/OpenACC listings (Listings 1–8). The pseudocode, Algorithm 2 in particular, is close to OpenACC but not quite. After the real OpenACC Fortran has been shown, this is confusing, and it reads like the listing repeated with annotations. The two also use different notation (index names, ρ_air vs `airmass_now`, `W` vs `total_weight`). Please either use standard, high-level pseudocode that clearly abstracts from the implementation, or drop it in favour of the listings, and use one notation throughout.
Related to this, please confirm with compiler feedback (`-Minfo=accel`) or a profiler how the collapsed `jc`/`k` iteration space in Listing 3 maps to gangs and vector lanes, and state the resulting access pattern for `p_tracer_now` and `total_weight`. Table 3 claims stride-1 access, but the paper does not show this.
### 3. Energy and energy–delay product
I welcome that the authors report energy alongside time to solution and compute an energy–delay product (EDP). This is still rare in model-development papers. The presentation should be tightened, however:
- **Like-for-like comparison.** The EDP reductions of 48–67% on 2–5 GPU nodes share the problem of major comment 1(a): they compare multi-node GPU runs with a single CPU node that was never scaled. The like-for-like single-node result, a 20% lower ED
P at 54% higher energy to solution, is the fair headline, and it is what the abstract should report.
- **Wording.** Energy to solution is higher on GPU at every node count. Because EDP rewards speed by construction, a lower EDP alongside higher energy does not demonstrate "superior computational efficiency" (Fig. 2 caption; similarly L1012–1013). Pl
ease state the trade-off plainly.
- **Measurement method.** The paper does not say how energy was measured. According to the data archive, it is node-level energy from the CSCS `jobreport` tool, integrating full-node power (CPU, GPU, memory and interconnect). Please state this in the
paper, and note that the CPU baseline therefore includes the power drawn by four idle GPUs.
- Module-only energy, if available, would relate more directly to this work (cf. 1(c)). J·s is also more usual than kWh·s.
## Minor comments
### Implementation and presentation of Sects. 3–4
1. **Memory layout.** The discussion of strided access (L209–213) does not state the array allocation order, without which the strides cannot be followed. Sect. 4.2 does not say whether array dimensions were changed for the GPU or whether ICON's nati
ve layout was kept. The block index also sits in different positions: second-outermost in `p_tracer_now(jc,k,jb,τ)`, outermost in the new `total_weight(jc,k,nt,nc,jb)` and `tsf_now(jc,nt,nc,jb)`. Please discuss array layout together with loop orderin
g, explain the choice for the new arrays, and consider a diagram of the memory organisation.
2. **Vertical parallelism.** Please state explicitly that the vertical loop can be parallelised freely in these modules, since this is a key condition for the collapsed `jc`/`k` scheme. It does not hold for general column-based physics parametrisations in NWP codes, which have vertical dependencies (e.g. Clement et al., 2018).
3. **Sect. 4.3 listing.** The detailed initialisation listing adds little. Please shorten it or move it to an appendix. More generally, Sect. 6.2.4 and the end of 6.2 repeat 6.2.2, and Sects. 8.2–8.5 are generic, so the manuscript could be shortened substantially.
4. Listing 8 (L872) contains `IF (vprm_flux_type(nt) == 'resp')` inside the innermost vegetation-class loop of the VPRM ensemble kernel. That is a string comparison in a device kernel, which is what the paper says it removed (L10, L634–635, L1157).
5. Listing 7: the loop variable `k` runs over vegetation classes, which is otherwise used for the vertical dimension (with `i` denoting classes). Please explain this or rename the variable.
6. `total_weight` and `tsf_now`/`tsf_next` are allocated over all (tracer, category) pairs, not only active ones. Please give the device-memory footprint as a function of these dimensions, which would make Sect. 8.4 more concrete.
7. L1159–1160: the complexity is not reduced to O(N_timesteps). The kernels still loop over all active pairs; what is removed is the per-pair string search.
### Performance reporting
8. Host–device transfers are described as infrequent and small (L355, L396–398). Please quantify the transfer volume and time, e.g. as part of the module breakdown requested in major comment 1(c).
9. Sect. 5.6 / Eq. 25: the estimate uses 10⁵ cells, while Table 5 gives ∼3×10⁵. Stage 2 is multiplied by N_lev although VPRM updates only the lowest level. With so few FLOPs per cell the kernels are memory-bound, so achieved bandwidth would be more informative than a FLOP count.
10. The archived job script and namelists show a 24 h simulation with a 30 s timestep. Please state these in the paper, together with the output settings, and say whether I/O is included in the wall-clock times.
11. Table 6: please state the CPU cores per rank in the GPU runs.
12. The benchmark (1 tracer, 2 categories, 45 members) is far from the "hundreds of tracers" claimed in the abstract (L25) and conclusions (L1182). Please either add a run closer to that regime or limit the claims.
### Validation
13. Fig. 3: the colour scale for the differences is ±10%, but the differences are said to be ≤ ±0.01%, so the panels show nothing. Please use an appropriate scale and add quantitative statistics (max/RMS, per tracer).
14. Please give the details of the probtest set-up (L1040–1051): configuration, length, fields and tracers tested, ensemble size, tolerances. The probtest configuration should also be included in the data archive (see minor 16).
### Code and data availability
15. The code availability statement (L1189–1191) says that the Zenodo archive of the ICON 2026.04 release includes the GPU-ported OEM and VPRM code. However, the `externals/art/` directory in that archive is empty. ART is only referenced in the recor
d's metadata, as a pinned commit (`fdac752`) of the public ART GitLab repository. That commit does contain the code described here, but a GitLab repository is not a persistent archive. Please archive the exact ICON-ART version used (e.g. on Zenodo),
cite it, and correct the statement. This would also resolve the version naming: the title refers to v2026.04, while ICON-ART is cited as version 2025.10 (Hoshyaripour et al., 2026).
16. The data archive (Hamzehloo, 2026) is well organised and includes the raw timing and energy data behind Fig. 2, together with the script that derives every plotted quantity. However, `README_figure2.md`, which `figure2_raw_data.csv` cites for the formulas, is missing. There are also no scripts or data for Fig. 3 or for the probtest configuration, although the record says it contains the post-processing scripts for the validation results. Please add these.