FZModules: A Heterogeneous Computing Framework for Customizable Scientific Data Compression Pipelines Conference: Workshops of the International Conference for High Performance Computing, Networking, Storage, and Analysis; November 16–21, 2025; St Louis, MO, USA Workshops of the International Confer
Skyler Ruiter, Jiannan Tian, Fengguang Song · 2025
Read FZModules: A Heterogeneous Computing Framework for Customizable Scientific Data Compression Pipelines Conference: Workshops of the International Conference for High Performance Computing, Networking, Storage, and Analysis; November 16–21, 2025; St Louis, MO, USA Workshops of the International Confer by Skyler Ruiter, Jiannan Tian, Fengguang Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20563.txt.
Introduction
Science runs make more data than disks can hold, so Skyler Ruiter, Jiannan Tian, and Fengguang Song build made-to-order squeeze pipelines in a 2025 conference paper. Lossy squeezing trades a small, bounded error for a much smaller file. The best pipeline shifts with each dataset and goal, yet chip-based squeezers hard-code their design and block quick trials. Their framework, FZModules, takes the other path.
Pipelines snap together from fast modules through a short, open interface. A task engine works out which stage needs what, moves memory, and runs stages side by side where it can. The team built three pipelines and tried them on four real science datasets. Speed matched the fused chip-based squeezers end to end. At the same time, size-versus-error quality matched higher-grade CPU and hybrid tools. The aim is fast, field-tuned design with no kernel rewrites.
Catalog record
| CREATOR | Skyler Ruiter, Jiannan Tian, Fengguang Song |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20563
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FZModules: A Heterogeneous Computing Framework for Customizable Scientific Data Compression Pipelines Conference: Workshops of the International Conference for High Performance Computing, Networking, Storage, and Analysis; November 16–21, 2025; St Louis, MO, USA Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC Workshops ’25), November 16–21, 2025, St Louis, MO, USA DOI: 10.1145/3731599.3767376 ISBN: 979-8-4007-1871-7/2025/11 CCS: Information systems Compression strategies CCS: Software and its engineering Frameworks Skyler Ruiter, Jiannan Tian, Fengguang Song Abstract . Modern scientific simulations and instruments generate data volumes that overwhelm memory and storage, throttling scalability. Lossy compression mitigates this by trading controlled error for reduced footprint and throughput gains, yet optimal pipelines are highly data and objective specific, demanding compression expertise. GPU compressors supply raw throughput but often hard - code fused kernels that hinder rapid experimentation, and underperform in rate–distortion. We present FZModules, a heterogeneous framework for assembling error - bounded custom compression pipelines from high - performance modules through a concise extensible interface. We further utilize an asynchronous task-backed execution library that infers data dependencies, manages memory movement, and exposes branch and stage level concurrency for powerful asynchronous compression pipelines. Evaluating three pipelines built with FZModules on four representative scientific datasets, we show they can compare end - to - end speedup of fused - kernel GPU compressors while achieving similar rate–distortion to higher fidelity CPU or hybrid compressors, enabling rapid, domain-tailored design. † † cc-license: by 1. Introduction Nowadays, large-scale scientific applications and advanced experimental instruments produce extreme-scale data for post hoc analysis and poses unprecedented complexity in data-processing workflows. This leads to not only a high pressure onto supercomputing subsystems (storage, memory, I/O, and communication), but also performance challenges when real-time and inlining processing are attempted. For example, Hardware/Hybrid Accelerated Cosmology Code (HACC) ( Habib et al. 2016 ) can produce data at an order of magnitude of petabytes in hundred-snapshot simulation of one trillion particles. At the same time, the “sink” to receive data, at the moment of data generation, faces overloads due to the excessive data stream-in. To this end, data reduction is becoming an effective method to address the big data issues. Previous work focuses on building and optimizing data reduction via error-controllable lossy compression as it ensures acceptable reconstruction quality while drastically reducing data size, making it commonly considered for scientific data reduction ( Tian et al. 2020 ; Liang et al. 2023 ; Zhao et al. 2020 ) . In many scientific applications, however, CPU based compressors can’t reach sufficiently high throughput to be used in these data processing workflows, such as SZ3 ( Liang et al. 2023 ) which as shown in prior work ( Fallin et al. 2025 ) , achieves tens of gigabytes per second with OpenMP acceleration, leading to the development of GPU accelerated compressors ( Tian et al. 2020 ; Zhang et al. 2023 ; Huang et al. 2023 ; Huang et al. 2024 ; Lindstrom 2014 ) , which often achieve performance an order of magnitude higher than CPU compressors. On the other hand, when fixing the global tolerable error (“error bound”), while GPU based compressors achieve high throughput, they usually achieve suboptimal rate-distortion performance due to the lower prediction accuracy of high-throughput predictors ( Zhao et al. 2021a ) . By contrast, CPU’s high-quality predictors involves cascadingly dependent profiling and sampling, hindering the data parallelism and the subsequent high throughput. This gap between the throughput and prediction accuracy had led to considerable work to close, such as cuSZ - i ( Liu et al. 2024b ) , which adapts a lightweight sampling-and-profiling method to optimize the prediction (inspired by QoZ ( Liu et al. 2024a ) ), and PFPL ( Fallin et al. 2025 ) , which implements a portable compressor that can achieve an order of magnitude higher throughput than SZ3 with its OpenMP version. With the wide range of data-reduction techniques at use and varying hardware constraints, constructing an optimized compressor to meet the compression ratio, data quality, and throughput for an application requires significant time and effort, as there is no universal best-fit compressor even for a single dataset. Therefore, a modular framework to compose, tailor, and test high-performance compression pipelines is required to facilitate the discovery of powerful custom compressors which meet the performance and quality needs of an application. In this paper, we present a heterogeneous framework for composing high-performance data reduction modules, FZModules. Built from cuSZ ( Tian et al. 2020 ) , this expanded framework allows for the rapid development of custom compression pipelines to meet the varied needs of each application and dataset. Specifically, we highlight three pipelines based on previous work: (1) FZMod-Default , which adapts a hybrid CPU-GPU design consisting of a GPU-accelerated Lorenzo predictor followed by CPU-based Huffman encoding, (2) FZMod-Speed , which swaps out the slower Huffman encoder for the efficient bit-shuffle and dictionary encoding of FZ-GPU ( Zhang et al. 2023 ) , and (3) FZMod-Quality , which removes the Lorenzo predictor in favor of the G-Interp algorithm found in cuSZ - i ( Liu et al. 2024b ) for better data prediction. We also support the use of a secondary lossless encoder for further compression, with ZStd ( zst [n. d.] ) being the currently supported secondary encoder module. In addition, we experimentally support the rapid creation of compression pipelines utilizing CUDASTF ( Augonnet et al. 2024 ) . This library allows for the description of tasks based on data dependencies, deriving a directed acyclic graph to dynamically manage asynchronous execution of tasks and memory management over the GPU and CPU. By utilizing CUDASTF we can leverage task-level concurrency for compression stages not data dependent on each other. FZMod-Default for example, when using its CUDASTF-enabled decompression pipeline, can populate outlier data onto the GPU while the CPU does Huffman decoding on the quantization codes, leveraging the heterogeneous nature of the computing environment. Our contributions are summarized as follows. • We design FZModules, an extensible framework for creating customized error-bounded lossy compressors to meet specific application requirements of speed and reconstruction quality. This framework takes into account the scope of heterogeneous computing systems that make up modern scientific applications to provide high end-to-end speedup while allowing for rapid testing of multiple pipelines. • We utilize CUDASTF in creating an experimental compression pipeline constructor, which allows for module creation to focus on data needs and high performance. The CUDASTF pipeline automatically leverages derived task concurrency and memory management between tasks, simplifying development and demonstrating the ability of compressors to leverage asynchronous execution on heterogeneous systems. • We evaluate three FZModules compression pipelines on four real-world application datasets from the Scientific Data Reduction Benchmarks ( Zhao et al. 2020 ) . We compare these pipelines against state-of-the-art GPU and CPU based compressors to show our library’s ability to attain comparable end-to-end speedup against pure-GPU compressors while also having high rate-distortion performance. FZModules is maintained on GitHub. 1 1 1 Repository: https://github.com/szcompressor/FZModules We organize the rest of this paper as follows: § 2 discusses related work. In § 3 we present the design, pipeline construction, and CUDASTF functionality. In § 4 we present the evaluation results for the comparisons between three FZMod pipelines and state-of-the-art CPU