Preprints
https://doi.org/10.5194/egusphere-2023-2547
https://doi.org/10.5194/egusphere-2023-2547
08 Jan 2024
 | 08 Jan 2024
Status: this preprint is open for discussion.

Accelerating Lagrangian transport simulations on graphics processing units: performance optimizations of MPTRAC v2.6

Lars Hoffmann, Kaveh Haghighi Mood, Andreas Herten, Markus Hrywniak, Jiri Kraus, Jan Clemens, and Mingzhao Liu

Abstract. Lagrangian particle dispersion models are indispensable tools for the study of atmospheric transport processes. However, Lagrangian transport simulations can become numerically expensive when large numbers of air parcels are involved. To accelerate these simulations, we made considerable efforts to port the Massive-Parallel Trajectory Calculations (MPTRAC) model to graphics processing units (GPUs). Here we discuss performance optimizations of the major bottleneck of the GPU code of MPTRAC, the advection kernel. Timeline, roofline, and memory analyses of the baseline GPU code revealed that the application is memory-bound and performance suffers from near-random memory access patterns. By changing the data structure of the horizontal wind and vertical velocity fields of the global meteorological data driving the simulations from Structure of Arrays (SoA) to Array of Structures (AoS), and by introducing a sorting method for better memory alignment of the particle data, performance was greatly improved. We evaluated the performance on NVIDIA A100 GPUs of the Jülich Wizard for European Leadership Science (JUWELS) Booster module at the Jülich Supercomputing Center, Germany. For our largest test case, transport simulations with 108 particles driven by the European Centre for Medium-Range Weather Forecasts (ECMWF) ERA5 reanalysis, we found that the runtime for the full set of physics computations was reduced by 75 %, including a reduction of 85 % for the advection kernel. In addition to demonstrating the benefits of code optimization for GPUs, we show that the runtime of CPU-only simulations is also improved. For our largest test case, we found a runtime reduction of 34 % for the physics computations, including a reduction of 65 % for the advection kernel. The code optimizations discussed here bring the MPTRAC model closer to applications on upcoming exascale high performance computing systems, and will also be of interest for optimizing the performance of other models using particle methods.

Lars Hoffmann, Kaveh Haghighi Mood, Andreas Herten, Markus Hrywniak, Jiri Kraus, Jan Clemens, and Mingzhao Liu

Status: open (until 04 Mar 2024)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse
  • RC1: 'Comment on egusphere-2023-2547', Anonymous Referee #1, 01 Feb 2024 reply
  • RC2: 'Comment on egusphere-2023-2547', Anonymous Referee #2, 03 Feb 2024 reply
  • RC3: 'Comment on egusphere-2023-2547', Anonymous Referee #3, 04 Feb 2024 reply
Lars Hoffmann, Kaveh Haghighi Mood, Andreas Herten, Markus Hrywniak, Jiri Kraus, Jan Clemens, and Mingzhao Liu

Data sets

Supplementary material to `Accelerating Lagrangian transport simulations on graphics processing units: performance optimizations of MPTRAC v2.6' Lars Hoffmann https://doi.org/10.5281/zenodo.10065785

Model code and software

Massive-Parallel Trajectory Calculations (MPTRAC) v2.6 L. Hoffmann et al. https://doi.org/10.5281/zenodo.10067751

Lars Hoffmann, Kaveh Haghighi Mood, Andreas Herten, Markus Hrywniak, Jiri Kraus, Jan Clemens, and Mingzhao Liu

Viewed

Total article views: 268 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
212 44 12 268 4 3
  • HTML: 212
  • PDF: 44
  • XML: 12
  • Total: 268
  • BibTeX: 4
  • EndNote: 3
Views and downloads (calculated since 08 Jan 2024)
Cumulative views and downloads (calculated since 08 Jan 2024)

Viewed (geographical distribution)

Total article views: 265 (including HTML, PDF, and XML) Thereof 265 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Latest update: 25 Feb 2024
Download
Short summary
Lagrangian particle dispersion models are crucial for studying atmospheric transport, but they can be computationally intensive. To speed up simulations, the MPTRAC model was adapted for GPUs. Performance optimizations of data structures and memory alignment resulted in run-time improvements of up to 75 % on NVIDIA A100 GPUs for ERA5-based simulations with 100 million particles. These optimizations make the MPTRAC model well suited for upcoming HPC systems.