EDPC: Accelerating Lossless Compression via Lightweight Probability Models and Decoupled Parallel Dataflow Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25),
Zeyi Lu, Xiaoxiao Ma, Yujun Huang, Minxiao Chen, Bin Chen, Baoyi An, Shu-Tao Xia · 2025
Read EDPC: Accelerating Lossless Compression via Lightweight Probability Models and Decoupled Parallel Dataflow Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), by Zeyi Lu, Xiaoxiao Ma, Yujun Huang, Minxiao Chen, Bin Chen, Baoyi An, Shu-Tao Xia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-18969.txt.
Introduction
Zeyi Lu and co-authors present a faster lossless compression system in a 2025 paper. The catalog title carries stray conference text, but the text itself names the system EDPC and states its aim plainly: shrink files with no loss, at speeds fit for live use. Growth in media data strains storage and bandwidth.
The design improves two layers at once. A new scoring metric guides a block that models links between bytes more finely. A second engine packs inner traits tightly, while a split pipeline removes waits between encoding and decoding steps. Tests showed compression about 2.7 times faster than prior methods, with a slightly better shrink ratio. The gains suit real-time work on large media streams.
Catalog record
| CREATOR | Zeyi Lu, Xiaoxiao Ma, Yujun Huang, Minxiao Chen, Bin Chen, Baoyi An, Shu-Tao Xia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-18969
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EDPC: Accelerating Lossless Compression via Lightweight Probability Models and Decoupled Parallel Dataflow Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3754739 ISBN: 979-8-4007-2035-2/2025/10 376 CCS: Information systems Information storage systems Zeyi Lu, Xiaoxiao Ma, Yujun Huang, Minxiao Chen, Bin Chen, Baoyi An, Shu-Tao Xia Abstract . The explosive growth of multi-source multimedia data has significantly increased the demands for transmission and storage, placing substantial pressure on bandwidth and storage infrastructures. While Autoregressive Compression Models (ACMs) have markedly improved compression efficiency through probabilistic prediction, current approaches remain constrained by two critical limitations: suboptimal compression ratios due to insufficient fine-grained feature extraction during probability modeling, and real-time processing bottlenecks caused by high resource consumption and low compression speeds. To address these challenges, we propose Efficient Dual-path Parallel Compression (EDPC), a hierarchically optimized compression framework that synergistically enhances modeling capability and execution efficiency via coordinated dual-path operations. At the modeling level, we introduce the Information Flow Refinement (IFR) metric grounded in mutual information theory, and design a Multi-path Byte Refinement Block (MBRB) to strengthen cross-byte dependency modeling via heterogeneous feature propagation. At the system level, we develop a Latent Transformation Engine (LTE) for compact high-dimensional feature representation and a Decoupled Pipeline Compression Architecture (DPCA) to eliminate encoding-decoding latency through pipelined parallelization. Experimental results demonstrate that EDPC achieves comprehensive improvements over state-of-the-art methods, including a 2.7× faster compression speed, and a 3.2% higher compression ratio. These advancements establish EDPC as an efficient solution for real-time processing of large-scale multimedia data in bandwidth-constrained scenarios. Our code is available at https://github.com/Magie0/EDPC . 1. Introduction Figure 1. We propose EDPC , an efficient dual-path parallel compression framework. At the modeling level , previous methods typically rely on single-path architectures, which restrict the diversity of information flow; in contrast, EDPC mitigates this bottleneck via a multi-branch design. At the system level , existing approaches struggle with high memory usage and underutilized hardware resources. Our framework employs latent transformation and pipelined parallelism to boost efficiency, enabling GPU-CPU collaboration. With the rapid development of 5G and cloud computing, the need to store and transmit massive volumes of high-resolution multimedia data in real time has become more pressing ( Xu et al. 2020 ; Zhang et al. 2024 ; Cao et al. 2024 ; Yan et al. 2024 ; Chen et al. 2025 ) . Traditional compression methods (e.g., Gzip ( Deutsch 1996 ) , Zstandard ( Collet 2016 ) ) often fail to achieve desirable compression ratios for increasingly multimodal data. Autoregressive Compression Models (ACMs) alleviate these limitations by iteratively predicting the probability distribution of each byte conditioned on previously encoded bytes, significantly outperforming conventional methods in terms of compression ratio. Yet, current ACMs still face two major obstacles. First , the information flow bottleneck constrains representational power. Most ACMs (e.g., TRACE ( Mao et al. 2022b ) , OREO ( Mao et al. 2022a ) , PAC ( Mao et al. 2023 ) ) adopt a single-branch design within their feature extraction modules, forcing all context information through a single path. This limits the capacity to capture diverse byte-level features and often leads to redundant representations. Although naively increasing the parameter count can improve modeling flexibility to some extent, it does not fundamentally address the single-path limitation. Second , current ACMs often exhibit inefficient resource utilization , especially for large-scale tasks such as real-time video streaming or high-resolution image transmission. For instance, NNCP ( Bellard 2019 ) leverages a Transformer with millions of parameters; however, its compression speed is insufficient for real-time deployment and demands excessive GPU memory. Although approaches like PAC offer partial relief, they still face significant constraints in terms of speed and memory usage. These challenges become prohibitive in scenarios that require continuous, large-scale data processing. To tackle these issues, we propose EDPC ( E fficient D ual-path P arallel C ompression), an integrated framework (see Figure 1 ) that unifies new byte-level modeling and an efficiency system design: • Multi-path Byte Refinement Block (MBRB). We adopt a dual-branch feature extraction scheme to overcome the information-flow limitations of single-path architectures. To quantify how additional branches contribute unique information, we propose an Information Flow Refinement (IFR) metric based on mutual information. Empirically, a two-branch structure offers the best trade-off between enriched feature diversity and computational overhead. • Latent Transformation Engine (LTE). To drastically reduce memory and computation, we project high-dimensional features into a lower-dimensional latent space and then reconstruct them with minimal information loss. Unlike a simple dimensionality-reduction layer, LTE employs a learnable Feature Distribution Matrix in the latent space, capturing critical inter-feature correlations. This design substantially cuts resource consumption while preserving crucial context. • Decoupled Pipeline Compression Architecture (DPCA). Conventional ACM pipelines typically serialize probability prediction and encoding, underutilizing modern hardware. Our design decouples these stages into a concurrent pipeline, allowing GPU-based prediction and CPU-based encoding to operate in parallel. Further, we split data streams into segments for multi-process encoding, maximizing throughput by exploiting heterogeneous hardware resources. Extensive evaluations on large-scale datasets show that EDPC significantly outperforms previous approaches: it reduces the parameter count by up to 4 \times , cuts GPU memory usage nearly in half, accelerates compression speed by a factor of 2.7, and achieves a 3.2% higher compression ratio compared to advanced baselines. In essence, EDPC’s multi-branch representation, latent transformation, and parallel execution work in tandem to enhance byte-level modeling while lowering system-level overhead. As a result, EDPC is ideally positioned for large-scale multimedia data scenarios, meeting the dual needs of high compression quality and efficient real-time processing. Our contributions are summarized as follows: (1) We propose an IFR metric to measure the unique information contributed by each branch and design the MBRB module to enhance byte-level feature representations significantly. (2) We introduce the LTE architecture, compressing features into a lower-dimensional latent space and leveraging a learnable Feature Distribution Matrix to capture context efficiently, thereby cutting memory overhead. (3) We develop the DPCA pipeline, which decouples probability prediction from encoding to enable parallel GPU-CPU execution, dramatically improving compression speed. (4) We integrate these innovations into the EDPC framework. Experiments show that EDPC not only achieves superior compression ratios but also significantly reduces GPU memory usage and encoding latency, offering an effective solution for large-scale, real-time data compression. 2. Related Work 2.1. Autoregressive Compression Model (ACM) Probability Estimation. In the current ACM, given a byte stream X=\{x_{1},x_{2},\ldots,x_{len}\} , the model initially encodes the first t bytes with equal probability. Then, starting from the (t+1) -th byte, the model uses the previous t bytes as input and calculates the conditional probability distribution for the i -th byte x_{i} through a forward pass ( Valmeekam et al. 2023 ; Knoll 2016 ; Huang et al. 2023 ; Mao et al. 2025a ) , (1) \Pr(i)=P(x_{i}\mid x_{i-t},\ldots,x_{i-1}) This process continues iteratively until all bytes are processed. At each step, the cross-entropy loss is calculated for the prediction (2) \text{loss}=\text{CrossEntropy}(\Pr(i),x_{i}) and gradients are computed via backpropagation, followed by updates to the model parameters ( Delétang et al. 2024 ; Ma et al. 2025b ; Liu et al. 2024 ; Shannon 1948a ) . Data