BOA Constrictor: A Mamba-based lossless compressor for High Energy Physics data

Akshat Gupta, Caterina Doglioni, Thomas J. Elliott · 2025

Read BOA Constrictor: A Mamba-based lossless compressor for High Energy Physics data by Akshat Gupta, Caterina Doglioni, Thomas J. Elliott (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11337.txt.

Introduction

Akshat Gupta, Caterina Doglioni, and Thomas Joseph Elliott present BOA Constrictor, a lossless compressor for high-energy physics data built on the Mamba sequence architecture. Collider experiments produce petabyte-scale data each year, and standard compressors such as LZMA leave deep structure in scientific data unexploited. Their system predicts the next byte with an autoregressive Mamba model and codes the stream in parallel as it flows.

On three structured physics data sets, BOA beat the strongest LZMA setting every time, with the largest gain on the most structured set. The model itself is only a few megabytes. The trade-off is speed: the prototype's storage-saving rate per unit time still lags behind.

Catalog record

CREATORAkshat Gupta, Caterina Doglioni, Thomas J. Elliott
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11337

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BOA Constrictor: A Mamba-based lossless compressor for High Energy Physics data Akshat Gupta, Caterina Doglioni, Thomas J. Elliott Abstract The petabyte-scale data generated annually by High Energy Physics (HEP) experiments like those at the Large Hadron Collider present a significant data storage challenge. Whilst traditional algorithms like LZMA and ZLIB are widely used, they often fail to exploit the deep structure inherent in scientific data. We investigate the application of modern state space models (SSMs) to this problem, which have shown promise for capturing long-range dependencies in sequences. We present the Bytewise Online Autoregressive (BOA) Constrictor, a novel, streaming-capable lossless compressor built upon the Mamba architecture. BOA combines an autoregressive Mamba model for next-byte prediction with a parallelised streaming range coder. We evaluate our method on three distinct structured datasets in HEP, demonstrating state-of-the-art compression ratios, improving upon LZMA-9 across all datasets. These improvements range from 2.21 × \times (vs. 1.69 × \times ) on the ATLAS dataset to a substantial 44.14 × \times (vs. 27.14 × \times ) on the highly-structured CMS dataset, with a modest ∼ 4.5 \sim 4.5 MB model size. However, this gain in compression ratio comes with a trade-off in throughput; the Storage-Saving Rate ( σ S ​ S ​ R \sigma_{SSR} ) of our prototype currently lags behind highly-optimised CPU-based algorithms like ZLIB. We conclude that while this Mamba-based approach is a highly promising proof-of-principle, significant future work on performance optimisation and hardware portability is required to develop it into a production-ready tool for the HEP community. INTRODUCTION As global demand for data storage and sharing continue to grow, managing such volumes has become increasingly challenging. This issue is particularly acute in high-energy physics (HEP), where vast and complex datasets routinely push the limits of existing compression and storage technologies: each year, experiments at the Large Hadron Collider (LHC) at CERN produce approximately thirty petabytes of data 10 . Current solutions, such as the ROOT framework combined with algorithms like Lempel–Ziv–Markov chain Algorithm (LZMA) and ZLIB, are currently used to address these challenges 8 ; 30 . In this work, we investigate whether greater gains in storage efficiency can be achieved within the constraints of existing infrastructure with a focus on minimal data deterioration and loss 23 , through improved lossless compression methods. Algorithms commonly employed for HEP data compression, such as LZMA and ZLIB, largely overlook the inherent structure of the data, including correlations between sequential measurements and dependencies among physical parameters 30 . A more “data-aware” compression can be achieved via the use of neural networks that learn the underlying patterns of data 24 ; 25 ; 29 ; 4 . While this is not a new approach, early neural compressors based on Recurrent Neural Networks (RNNs) were computationally prohibitive. Later models based on the Transformer architecture showed very good performance but suffer from a quadratic complexity bottleneck ( O(L^{2}) ) 35 ; 32 ; 38 . The recent development of the Mamba State Space Model (SSM) 14 ; 16 has fundamentally changed this landscape. Mamba combines the strengths of RNNs and Transformers, offering a model that can process sequences with linear time complexity and constant memory during inference 14 ; 16 . They can also capture extremely long-range dependencies 14 ; 16 . This unique combination of features makes them a prime-candidate for a high-performance learned compressor. This work presents the design, implementation, and evaluation of the Bytewise Online Autoregressive (BOA) Constrictor, a streaming lossless compressor built on the Mamba architecture. A modest \sim 4.5 MB BOA model coupled to a parallelised range coder consistently outperforms LZMA-9 on three representative HEP datasets: from 1.69 \times to 2.21 \times on ATLAS and from 27.14 \times to 44.14 \times on CMS. This improved compression comes at the expense of lower throughput on current hardware, highlighting the deployment trade-offs for neural compressors in HEP. Theoretical Framework Entropy-Based Compression and Cross-Entropy Loss Consider data sourced from an independent and identically-distributed (i.i.d.) distribution P , taking possible values x\in\mathcal{X} . The entropy H(P)=-\sum_{x}P(x)\log_{256}P(x) 31 quantifies the fundamental limit of lossless compression, in bytes, in the asymptotic limit. By the source–coding theorem for stationary ergodic sources, the expected code length per symbol satisfies \tfrac{1}{n}\mathbb{E}[\ell(X_{1:n})]\geq H(P) bytes 31 , with equality achieved as n\to\infty under optimal coding that uses P 21 . For a specific sequence x , the prefix Kolmogorov complexity K(x)=\min_{p}\{|p|:U(p)=x\} implies that any lossless code C must satisfy |C(x)|\geq K(x)-O(1) , whilst \mathbb{E}[K(X_{1:n})]=nH(P)+O(1) 21 . When we approximate the true source distribution P with a model distribution Q , the achievable compression is governed by the cross-entropy H(P,Q)=H(P)+D_{\mathrm{KL}}(P\|Q) , where D_{\mathrm{KL}}(P\|Q)=\sum_{x}P(x)\log_{256}(P(x)/Q(x)) is the Kullback-Leibler Distance 22 . The optimal expected code length per symbol under Q is H(P,Q) bytes, reaching the minimum H(P) iff Q=P 22 . This information-theoretic perspective provides the foundation for our compression approach. We serialise each table row into a causal byte sequence \mathbf{y}=(y_{1},\ldots,y_{L})\in\{0,\ldots,255\}^{L} using a decoder-safe grammar G . An entropy-based compressor comprises two components: a predictor Q that assigns conditional probabilities Q(y_{t}=k\mid y_{<t}) over k\in\{0,\ldots,255\} , conditioned only on the preceding bytes; and an entropy coder C that transforms the pair (\{q_{t}\}_{t=1}^{L},\mathbf{y}) into a compressed bytestream, where q_{t}:=Q(y_{t}\mid y_{<t}) is the probability assigned to the realised byte. The ideal compression length (in bytes) under the true distribution P would be -\log_{256}P(y_{t}\mid y_{<t}) at each position. In practice, using the predictor Q , the actual compression length is \displaystyle\text{Total Bytes}\;=\;\sum_{t=1}^{L}-\log_{256}Q(y_{t}\mid y_{<t}). (1) That is, the average number of bytes per symbol is H(P,Q) , corresponding to the theoretical ideal H(P) iff Q=P . Traditional compressors approximate these probabilities using heuristics. Dictionary-based methods (e.g. LZMA 39 ; 40 ) implicitly assign high probability to sequences that have appeared previously in a sliding window, proving very effective at capturing literal and local redundancies. Statistical methods (e.g. Prediction by Partial Matching (PPM) 27 ; 12 ) explicitly build a statistical model of symbol contexts to predict the next symbol. In our approach, we employ a neural network trained as an autoregressive model to serve as the probability estimator Q . This work leverages the Mamba architecture as a state-of-the-art predictive model for this purpose. For byte tokens y_{t}\in\{0,\ldots,255\} with model probabilities p_{i}^{(t)}=Q(Y_{t}=i\mid Y_{<t}) , the per-step loss is \displaystyle\mathcal{L}_{\mathrm{CE},t}=-\log_{256}Q(y_{t}\mid y_{<t}). (2) The total sequence loss is \mathcal{L}_{\mathrm{CE}}=\sum_{t=1}^{L}\mathcal{L}_{\mathrm{CE},t} . Taking the expectation over the true source P , we obtain \displaystyle\mathbb{E}_{P}[\mathcal{L}_{\mathrm{CE}}=H(P,Q)] (3) \displaystyle=H(P)+D_{\mathrm{KL}}(P\!\parallel\!Q)\geq H(P), with equality iff Q=P . Minimising this cross-entropy loss therefore directly optimises our model for compression, as reducing D_{\mathrm{KL}}(P\|Q) simultaneously improves both predictive accuracy and compression efficiency. Range coding To implement the entropy-based compression, our framework pairs an autoregressive neural predictor ( Q ) with an entropy coder ( C ). For the coder, we use range coding 26 , which is an integer form of arithmetic coding that emits bytes as soon as they are fixed. It keeps an integer base L and width R so the active interval is [L,\,L{+}R) on a w -bit ring. Symbols are encoded using an integer Cumulative Distribution Function (CDF). For the predictor, we implement Q using the Mamba state-space model (SSM) 16 , chosen for its ability to model long-range dependencies with linear-time complexity and constant memory during inference (see Appendix A.

Read the complete text