Preprints
https://doi.org/10.5194/egusphere-2024-753
https://doi.org/10.5194/egusphere-2024-753
25 Apr 2024
 | 25 Apr 2024
Status: this preprint is open for discussion.

The effect of lossy compression of numerical weather prediction data on data analysis: a case study using enstools-compression 2023.11

Oriol Tintó Prims, Robert Redl, Marc Rautenhaus, Tobias Selz, Takumi Matsunobu, Kameswar Rao Modali, and George Craig

Abstract. The increasing amount of data in meteorological science requires effective data reduction methods. Our study demonstrates the use of advanced scientific lossy compression techniques to significantly reduce the size of these large datasets, achieving reductions ranging from 5x to over 150x, while ensuring data integrity is maintained. A key aspect of our work is the development of the 'enstools-compression' Python library. This user-friendly tool simplifies the application of lossy compression for Earth scientists and is integrated into the commonly used NetCDF file format workflows in atmospheric sciences. Based on the HDF5 compression filter architecture, 'enstools-compression' is easily used in Python scripts or via command line, enhancing its accessibility for the scientific community. A series of examples, drawn from current atmospheric science research, shows how lossy compression can efficiently manage large meteorological datasets while maintaining a balance between reducing data size and preserving scientific accuracy. This work addresses the challenge of making lossy compression more accessible, marking a significant step forward in efficient data handling in Earth sciences.

Publisher's note: Copernicus Publications remains neutral with regard to jurisdictional claims made in the text, published maps, institutional affiliations, or any other geographical representation in this preprint. The responsibility to include appropriate place names lies with the authors.
Oriol Tintó Prims, Robert Redl, Marc Rautenhaus, Tobias Selz, Takumi Matsunobu, Kameswar Rao Modali, and George Craig

Status: open (until 20 Jun 2024)

Comment types: AC – author | RC – referee | CC – community | EC – editor | CEC – chief editor | : Report abuse
Oriol Tintó Prims, Robert Redl, Marc Rautenhaus, Tobias Selz, Takumi Matsunobu, Kameswar Rao Modali, and George Craig

Model code and software

enstools-compression Oriol Tintó and Robert Redl https://doi.org/10.5281/zenodo.10998676

Interactive computing environment

The effect of lossy compression of numerical weather prediction data on data analysis: software to reproduce figures using enstools-compression Oriol Tintó Prims https://doi.org/10.5281/zenodo.10998604

Oriol Tintó Prims, Robert Redl, Marc Rautenhaus, Tobias Selz, Takumi Matsunobu, Kameswar Rao Modali, and George Craig

Viewed

Total article views: 206 (including HTML, PDF, and XML)
HTML PDF XML Total BibTeX EndNote
151 48 7 206 4 4
  • HTML: 151
  • PDF: 48
  • XML: 7
  • Total: 206
  • BibTeX: 4
  • EndNote: 4
Views and downloads (calculated since 25 Apr 2024)
Cumulative views and downloads (calculated since 25 Apr 2024)

Viewed (geographical distribution)

Total article views: 204 (including HTML, PDF, and XML) Thereof 204 with geography defined and 0 with unknown origin.
Country # Views %
  • 1
1
 
 
 
 
Latest update: 20 May 2024
Download
Short summary
Advanced compression techniques can drastically reduce the size of meteorological datasets (by 5x to 150x) without compromising the data's scientific value. We developed a user-friendly tool called 'enstools-compression' that makes this compression simple for Earth scientists. This tool works seamlessly with common weather and climate data formats. Our work shows that lossy compression can significantly improve how researchers store and analyze large meteorological datasets.