Error bounded compression for weather and climate applications
2025
Read Error bounded compression for weather and climate applications (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22265.txt.
Introduction
Langwen Huang leads an eight-author team compressing weather and climate data that is outgrowing its storage. Simulations at rising resolution pour out petabytes. Their compressor, EBCC, squeezes hard while promising that no value is off by more than a stated error bound, the guarantee scientists need before trusting a shrunken archive.
Version two layers two techniques: a JPEG2000 base layer captures the bulk of the field cheaply. A sparse wavelet residual layer hunts down the long tail of extreme errors the base layer leaves behind. Three-dimensional chunks are handled through a multicomponent transform. A rate allocation step aware of the error bound picks coding effort to nearly meet the target while trading size against average error wisely. Benchmarks against established compressors on weather-relevant data carry the evaluation.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-22265
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Error bounded compression for weather and climate applications Abstract As the resolution of weather and climate simulations increases, the amount of data produced is growing rapidly from hundreds of terabytes to tens of petabytes. The huge size becomes a limiting factor for broader adoption, and its fast growth rate will soon exhaust all the available storage devices. To address these issues, we present EBCC (Error Bounded Climate-data Compressor). It follows a two-layer compression approach: a base compression layer using JPEG2000 to capture the bulk of the data with a high compression ratio, and a residual compression layer using wavelet transform and SPIHT (Set Partitioning In Hierarchical Trees) encoding to efficiently eliminate long-tail extreme errors introduced by the base compression layer. It incorporates a feedback rate-control mechanism for both layers that adjusts compression ratios to achieve the specified maximum error target. We evaluate EBCC alongside other established error-bounded compression methods on several benchmarks related to weather and climate science. The benchmarks include error statistics, a case study on primitive and derived variables near a tropical cyclone, evaluation of the closure of the global energy budget, and a Lagrangian air parcel trajectory simulation. This is the first time that trajectory simulation is used to benchmark compression methods. According to the histogram of the compression errors, our method concentrates most errors near zero, while other error-bounded methods tend to distribute errors uniformly within the error bound. EBCC outperforms other methods in the benchmarks at relative error targets ranging from 0.1% to 10% and achieves compression ratios from 15 × \times to more than 300 × \times , respectively. In the energy budget closure and Lagrangian trajectory benchmarks, it can achieve more than 100 × \times compression while keeping errors within natural variability derived from ERA5 uncertainty members. This verifies the effectiveness of EBCC in creating heavily compressed weather and climate datasets suitable for downstream applications. The source code of EBCC is available in github.com/spcl/EBCC . \Author [1][[email protected]]LangwenHuang \Author [1, 4]LuigiFusco \Author [1]FlorianScheidl \Author [2]JanZibell \Author [2]Michael ArmandSprenger \Author [3]SebastianSchemm \Author [1]TorstenHoefler 1]Scalable Parallel Computing Laboratory, Department of Computer Science, ETH Zurich, Switzerland 2]Institute for Atmospheric and Climate Science, Department of Environmental Systems Science, ETH Zurich, Switzerland 3]Department of Applied Mathematics and Theoretical Physics, Centre for Mathematical Sciences, University of Cambridge, Cambridge, UK 4]Microsoft Ibérica, Madrid, Spain † † firstpage: 1 \introduction Numerical weather and climate simulations produce a huge amount of data ranging from hundreds of terabytes to tens of petabytes ( Bauer et al. 2021 ; Acosta et al. 2024 ; Govett et al. 2024 ) . Analyzing these data requires significant storage and computing resources, which are only available to a few researchers. Moreover, the volume of data is growing rapidly as higher-resolution simulations are required to address the challenges posed by climate change, particularly extreme weather events. As the growth rate of the simulation data eclipses that of storage capacity, a shortage of storage space becomes inevitable ( Hoefler et al. 2023 ; ECMWF 2023 ) . As a result, there is an urgent need for data compression to store high-resolution simulation data while ideally having fast data access. The exploration of data compression for weather and climate data starts with the lossless compression method ( Rice 1996 ; Lindstrom 2017 ; Huang et al. 2016 ) where data can be exactly restored after compression and decompression processes. However, they can hardly reach a high compression ratio above 6 \times for common atmospheric variables ( Mummadisetty et al. 2015 ; Huang et al. 2016 ) because they attempt to store the random noise component of the data, which is impossible to compress according to Shannon’s source coding theorem ( Shannon 1948 ) . In contrast, lossy compression allows for an approximation of the original data in exchange for significantly higher compression ratios. The simplest form of lossy compression is quantization, which represents a block of data with linearly scaled integers with lower number of bits than the original format. It is often used in conjunction with lossless compression methods to achieve higher compression ratios ( Silver and Zender 2017 ) . Due to its simplicity, it is widely supported in modern scientific data formats such as NetCDF-4 ( Unidata 2025 ) , HDF5 ( Folk et al. 1999 ) , GRIB2 ( WMO 2025 ) and Zarr ( Miles et al. 2025 ) . Bit grooming ( Zender 2016 ) takes another approach of quantization by “shaving” the least significant bits of floating-point values to zeros. Bit rounding ( Klöwer et al. 2021 ) extends this idea with the bit information criterion to determine the number of bits to shave off. Based on the quantization - lossless compression pipeline, transform-based methods such as JPEG ( Wallace 1991 ) , JPEG2000 ( ISO Central Secretary 2024 ) and SPECK ( Chao and Gray 2006 ) add a reversible transform before the quantization step to further increase the compression ratio. The SPERR ( Li et al. 2023 ) compressor extends SPECK with a outlier encoding layer to bound the maximum compression error. Variational autoencoders ( Han et al. 2024 ; Mirowski et al. 2024 ) follow a similar pipeline as transform-based methods but replace the pre-defined transform with learned neural networks. Parallel to previous methods, prediction-based methods can achieve high compression ratios and max error bounded compression by estimating values from their neighbors and encoding the residuals. Examples include FPZIP ( Lindstrom 2017 ) , and the SZ family of methods ( Di and Cappello 2016 ; Liang et al. 2022 ; Jian et al. 2024 ) . In addition, it is also possible to compress multidimensional data via tensor decomposition ( Ballester-Ripoll et al. 2019 ) , or implicit neural representation ( Huang and Hoefler 2022 ) . Figure 1: Diagram of EBCC (Error Bounded Climate Compressor) method. It compresses weather & climate datasets using a two-layer approach ( section 1 ): the base compression layer uses JPEG2000 for bulk compression, while the residual compression layer encodes excessive errors in the residue using discrete wavelet transform (DWT) and the set partitioning in hierarchical tree (SPIHT) algorithm. Both layers incorporate feedback control to achieve the target error bound. EBCC is evaluated on a suite of benchmarks and outperforms existing compression methods ( section 2 ) Although lossy compression methods can achieve higher compression ratios than lossless ones, they introduce an additional complexity to the user due to the rate-distortion trade-off: users must decide a compression level where the resulting error metrics are acceptable for their applications. However, there is no consensus on the error metrics for weather and climate data. Point-wise metrics such as root-mean-square error (RMSE), peak signal-to-noise ratio (PSNR), and maximum absolute error are often used as the default metrics to evaluate compression errors. In addition, these metrics can be directly used as the error target in the compression methods. However, point-wise metrics oversimplify the compression errors as they ignore the correlation between neighboring data points. Baker et al. 2019 suggests using the structural similarity index (SSIM) to evaluate the visual quality of the compressed climate data, and proposes a SSIM target of 0.99995 indicating visually indistinguishable data according to field experts. More statistical and visual metrics are proposed in