STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis; November 16–21, 2025; St Louis, MO, USA The International Conference for High Performance Computing,
Daoce Wang, Pascal Grosset, Jesus Pulido, Jiannan Tian, Tushar M. Athawale, Jinda Jia, Baixi Sun, Boyuan Zhang, Sian Jin, Kai Zhao, James Ahrens, Fengguang Song · 2025
Read STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis; November 16–21, 2025; St Louis, MO, USA The International Conference for High Performance Computing, by Daoce Wang, Pascal Grosset, Jesus Pulido, Jiannan Tian, Tushar M. Athawale, Jinda Jia, Baixi Sun, Boyuan Zhang, Sian Jin, Kai Zhao, James Ahrens, Fengguang Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01626.txt.
Introduction
Scientific simulations produce data too vast to store or move easily, so error-bounded lossy compression, which shrinks data within a set error limit, is a leading remedy. Daoce Wang, with eleven co-authors, presents STZ, a streaming compression framework that keeps quality and speed high while adding flexible access. Progressive decompression, which refines data gradually, and random access, which reads any part on demand, usually degrade both.
The framework is described as the first to support both features at once. A hierarchical partitioning strategy enables streaming, and a matching prediction scheme offsets the quality cost of splitting the data, reaching quality comparable to a leading non-streaming compressor. Wang and the team also report high compression and decompression speeds, aiming at analysis workflows that need data on demand.
Catalog record
| CREATOR | Daoce Wang, Pascal Grosset, Jesus Pulido, Jiannan Tian, Tushar M. Athawale, Jinda Jia, Baixi Sun, Boyuan Zhang, Sian Jin, Kai Zhao, James Ahrens, Fengguang Song |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis; November 16–21, 2025; St Louis, MO, USA The International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’25), November 16–21, 2025, St Louis, MO, USA DOI: 10.1145/3712285.3759795 ISBN: 979-8-4007-1466-5/2025/11 Daoce Wang, Pascal Grosset, Jesus Pulido, Jiannan Tian, Tushar M. Athawale, Jinda Jia, Baixi Sun, Boyuan Zhang, Sian Jin, Kai Zhao, James Ahrens, Fengguang Song Abstract . Error-bounded lossy compression is one of the most efficient solutions to reduce the volume of scientific data. For lossy compression, progressive decompression and random-access decompression are critical features that enable on-demand data access and flexible analysis workflows. However, these features can severely degrade compression quality and speed. To address these limitations, we propose a novel streaming compression framework that supports both progressive decompression and random-access decompression while maintaining high compression quality and speed. Our contributions are three-fold: (1) we design the first compression framework that simultaneously enables both progressive decompression and random-access decompression; (2) we introduce a hierarchical partitioning strategy to enable both streaming features, along with a hierarchical prediction mechanism that mitigates the impact of partitioning and achieves high compression quality—even comparable to state-of-the-art (SOTA) non-streaming compressor SZ3; and (3) our framework delivers high compression and decompression speed, up to 6.7 × \times faster than SZ3. † † cc-license: by 1. Introduction Supercomputers have quickly approached exascale capabilities with increasingly higher computing power, causing the volume of scientific data storage and transmission to grow exponentially. Traditionally, a simulation with a resolution of 4096^{3} can generate approximately 5 TB of data per snapshot. Thus, running a single simulation with 200 snapshots would require a total disk storage capacity of 1 PB. Simulations in exascale scenarios are even larger—often involving thousands of points per axis—highlighting the necessity for effective data size reduction. While computing power has advanced rapidly, storage bandwidth has not kept pace, widening the gap between processing capability and data movement. To fill this gap, data reduction techniques have become critical for conserving storage bandwidth, reducing power consumption, and optimizing HPC system performance. Traditional lossless compression techniques, such as GZIP ( Deutsch 1996 ) and Zstandard ( Zstandard 2020 ) , typically achieve compression ratios of only up to 2 for scientific data. In contrast, new generations of lossy compressors—known as error-bounded lossy compressors—have been developed, including SZ ( Di and Cappello 2016 ; Tao et al. 2017 ; Liang et al. 2018 ; Zhao et al. 2021 ) , ZFP ( Lindstrom 2014 ) , MGARD ( Gong et al. 2023 ; Ainsworth et al. 2018 ) , SPERR ( Li et al. 2023 ) , and TTHRESH ( Ballester-Ripoll et al. 2020 ) . These compressors provide significantly higher compression ratios while minimizing quality loss in the reconstructed data and preserving its fidelity for post-analysis. Figure 1 . Iso-surface of original (top, 256\times 256\times 2048 , 1GB ) and down-sampled (bottom, 128\times 128\times 1024 , 128MB ) WarpX dataset. Visual differences between them are virtually imperceptible (SSIM=0.96), highlighting the need for progressive decompression. Although lossy compression can significantly reduce the storage and transmission overhead, traditional compression workflows require the entire compressed data to be fully decompressed for post-analysis or visualization, which restores the dataset to its original uncompressed size. However, many scientific use cases, such as inspecting regions of interest (e.g., 2D slices in a 3D dataset) or generating coarse-resolution previews ( Zgraggen et al. 2017 ; Ulmer et al. 2024 ) (as shown in Figure 1 ), do not require full data reconstruction. In these scenarios, decompressing and storing the entire dataset introduces significant computational, memory, I/O, and storage overhead. This problem is especially severe in two cases: (1) For analyzing terabyte-scale datasets on local devices, limited RAM and disk space can severely restrict the user’s ability to decompress and process the data. (2) For larger datasets, such as petabyte-scale datasets from extreme-scale simulations, the decompressed data can be too large to fit even in the memory of one or a few HPC nodes, making full decompression impractical on HPC systems. These challenges highlight the urgent need for on-demand decompression frameworks that support targeted access to compressed scientific data without reconstructing the entire dataset. Streaming compression is a method that supports both progressive decompression (i.e., decompressing data to a lower resolution) and random-access decompression (i.e., selectively decompressing parts of the data), offering an effective solution to these challenges. However, many scientific compressors, such as SZ3, do not support streaming compression, as it can significantly degrade compression quality and speed. Moreover, to the best of our knowledge, no existing scientific compressor supports both progressive decompression and random-access decompression simultaneously. For example, ZFP supports random-access decompression by partitioning data into 4^{3} blocks and processes each block independently during compression. However, this block-wise approach sacrifices compression quality/ratio due to the loss of spatial information between blocks (as summarized in Table 1 ). On the other hand, SPERR supports progressive decompression through its wavelet transform, which enables lower-resolution extraction while preserving global data structure. However, the wavelet transform used by SPERR is computationally expensive, resulting in slow performance and making it unsuitable for in-situ or online compression scenarios. Table 1 . Features of different compressors. SZ3 SPERR MGARD ZFP Ours Progressive ✗ ✓ ✓ ✗ ✓ Random Access ✗ ✗ ✗ ✓ ✓ Speed mid very low low very high high Comp. Quality high very high mid low high To address these limitations, we propose a novel streaming compression framework that fills the gap between streaming capabilities and high compression quality and speed. Our framework supports both progressive decompression and random-access decompression, enabling flexible HPC workflows for handling massive scientific data. We evaluate our solution against four state-of-the-art (SOTA) compressors and demonstrate that it enables streaming capabilities without compromising compression quality and speed. The features of our work and the baseline compressors are summarized in Table 1 , and detailed evaluation results are presented in § 4 . Our contributions are summarized as follows: 1. We propose a first-of-its-kind streaming compression framework that simultaneously supports progressive decompression and random-access decompression. 2. To enable streaming compression, we first introduce a hierarchical partitioning strategy. To mitigate compression quality loss caused by partitioning, we propose a hierarchical prediction method to preserve spatial information across different hierarchical levels, enabling our framework to achieve even comparable compression quality to the non-streaming compressor SZ3. 3. Our framework achieves high compression and decompression speeds, with high parallel efficiency, and is up to 6.7 \times faster than SOTA compressors with the same compression quality and ratio. 4. We evaluate our framework against four SOTA scientific lossy compressors: SZ3, SPERR, MGARD-X, and ZFP, using four real-world scientific datasets. The results show that our solution achieves the best overall trade-off between compression quality and speed. Notably, none of the SOTA compressors support both progressive decompression and random-access decompression. 2. Related Work 2.1. Lossy Compression for Scientific Data Most scientific simulations use floating-point data. There are two main categories of floating-point data compression: lossless and lossy compression. Lossless compression suffers from low compression ratios on scientific data (only up to 2 \times ( Chen et al. 2024