the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
The effect of lossy compression of numerical weather prediction data on data analysis: a case study using enstools-compression 2023.11
Abstract. The increasing amount of data in meteorological science requires effective data reduction methods. Our study demonstrates the use of advanced scientific lossy compression techniques to significantly reduce the size of these large datasets, achieving reductions ranging from 5x to over 150x, while ensuring data integrity is maintained. A key aspect of our work is the development of the 'enstools-compression' Python library. This user-friendly tool simplifies the application of lossy compression for Earth scientists and is integrated into the commonly used NetCDF file format workflows in atmospheric sciences. Based on the HDF5 compression filter architecture, 'enstools-compression' is easily used in Python scripts or via command line, enhancing its accessibility for the scientific community. A series of examples, drawn from current atmospheric science research, shows how lossy compression can efficiently manage large meteorological datasets while maintaining a balance between reducing data size and preserving scientific accuracy. This work addresses the challenge of making lossy compression more accessible, marking a significant step forward in efficient data handling in Earth sciences.
- Preprint
(4782 KB) - Metadata XML
- BibTeX
- EndNote
Status: open (until 20 Jun 2024)
Model code and software
enstools-compression Oriol Tintó and Robert Redl https://doi.org/10.5281/zenodo.10998676
Interactive computing environment
The effect of lossy compression of numerical weather prediction data on data analysis: software to reproduce figures using enstools-compression Oriol Tintó Prims https://doi.org/10.5281/zenodo.10998604
Viewed
HTML | XML | Total | BibTeX | EndNote | |
---|---|---|---|---|---|
115 | 36 | 7 | 158 | 3 | 3 |
- HTML: 115
- PDF: 36
- XML: 7
- Total: 158
- BibTeX: 3
- EndNote: 3
Viewed (geographical distribution)
Country | # | Views | % |
---|
Total: | 0 |
HTML: | 0 |
PDF: | 0 |
XML: | 0 |
- 1