OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data Yan Zhao, Zhengxue Cheng, Junxuan Zhang, Dajiang Zhou, Qunshan Gu, Qi Wang, Li Song Abstract Lossless compression is essential for efficient data storage and transmission. Although learning-based lossless compressors achieve strong results, most of them are designed for a single modality, leading to redundant compressor deployments in multi-modal settings. Designing a unified multi-modal compressor is critical yet challenging, as different data types vary largely in format, dimension, and statistics. Multi-modal large language models offer a promising resolution but remain too complex for practical use. Thus, we propose OmniZip , a unified and lightweight lossless compressor for multi-modal data (like image, text, speech, tactile, database, and gene sequence) . Built on a lightweight backbone, OmniZip incorporates three key components to enable efficient multi-modal lossless compression: a modality-unified tokenizer that reversibly transforms diverse data into tokens, a modality-routing context learning mechanism that enables flexible multi-modal context modeling, and a modality-routing feedforward design that further enhances the model’s nonlinear representation flexibility. A reparameterization training strategy is used to enhance model capacity. OmniZip outperforms or matches other state-of-the-art compressors on multiple modalities, achieving 42%, 57%, 62% and 42%, 53% higher compression efficiency than gzip on CLIC-M, TouchandGo, enwik9, LibriSpeech, and WikiSQL datasets, respectively. It also supports near real-time inference on resource-constrained edge devices, reaching about 1MB/s on MacBook CPUs and iPhone NPUs. Our code is released at https://github.com/adminasmi/OmniZip-CVPR2026 . 1 Introduction According to information theory 60 , the optimal code length for a symbol is determined by its negative logarithmic probability. This insight motivates the modern lossless compression strategy: use probabilistic models to estimate data likelihoods and apply entropy coding 34 ; 33 ; 26 to encode data. Recent approaches further leverage neural networks and large language models (LLMs) to improve probability estimation and enhance compression performance 22 ; 64 ; 9 ; 13 ; 71 ; 44 ; 46 ; 32 ; 67 ; 42 ; 24 ; 74 ; 75 ; 15 . Despite recent progress, learning-based lossless compression faces two key challenges. • First , many existing methods, especially those based on LLMs 64 ; 24 ; 13 ; 22 , suffer from excessive complexity. These models often contain billions of parameters, far exceeding the size of compressed data. Since compression speed is decided by the model’s inference speed, such complexity severely limits these methods’ practical deployments. For example, compressing a single 1080p image using LLaMA3-8B 3 takes over 30 minutes 24 ; 13 , while 8 ; 38 need several days to compress 1 GB of text. • Second, most existing methods are designed for a single modality 64 ; 5 ; 13 ; 71 , necessitating separate compressors in multi-modal systems. This increases software complexity and hardware costs. Despite its importance, multi-modal lossless compression remains challenging due to the heterogeneity of modalities: text is discrete and sequential, images are two-dimensional and spatially organized, speech is continuous with smooth spectral patterns, database contains structured categorical fields, and gene sequences exhibit symbolic structures with characteristic motifs. Some works 22 explore multi-modal lossless compression by converting all data to ASCII text, causing sub-optimal performance on non-text modalities. Figure 1 : Left : The world is multi-modal, motivating the need for unified lossless compression. Multi-modal compressors’ performance is shown across modalities, where points closer to the edge indicate better efficiency (lower bits/Byte). Right : Lightweight design ensures broad usability across platforms. OmniZip achieves near real-time inference (hundreds of KB/s to MB/s) even on edge devices. To tackle these challenges, we propose OmniZip , a unified and lightweight lossless compressor for multi-modal data. It supports most real-world data types suitable for lossless compression: image-like data (e.g., natural image, medical image, tactile signals), text-like data (e.g., natural language, gene sequence, database), and speech. Built upon a lightweight backbone, OmniZip explicitly addresses modality heterogeneity and enables efficient multi-modal compression through three key designs: (1) a modality-unified tokenizer that maps diverse data into a unified and fully reversible token space, (2) a modality-routing context learning mechanism that enables flexible multi-modal context modeling, and (3) a modality-routing feedforward process that introduces adaptive feedforward representations. A reparameterization training strategy further enhances the model capacity without increasing the inference cost. With much fewer parameters, OmniZip matches or surpasses the SOTA compressors across diverse modalities. It supports deployment across CPUs, NPUs, and GPUs, achieving near real-time inference even on resource-constrained edge devices (e.g., laptops and mobile phones). Generally, our contributions can be concluded as: • We propose OmniZip, a unified and lightweight lossless compressor for multi-modal data. It supports efficient lossless compression for image-like (natural image, medical image, tactile signal), text-like (natural language, gene sequence, database), and speech data. OmniZip explicitly addresses modality heterogeneity and provides useful insights for the emerging multi-modal applications. • Built on a lightweight backbone, OmniZip integrates three key techniques for efficient multi-modality lossless compression: modality-unified tokenization, modality-routing contextual learning, and modality-routing feedforward. A reparameterization training strategy is also employed to enhance the model’s capacity. • We evaluate OmniZip on 16 datasets across seven modalities. It outperforms or matches other SOTA compressors, achieving 42%, 57%, 62% and 42%, 53% higher compression efficiency than gzip on CLIC-M, TouchandGo, enwik9, LibriSpeech, and WikiSQL datasets. Even on edge devices like MacBook CPU or iPhone NPU, OmniZip reaches near real-time inference (0.1 \sim 1MB/s). 2 Related Work Classical Lossless Compressors. Classical lossless compression can be broadly categorized into general-purpose and modality-specific approaches. General-purpose compressors such as gzip 52 , bzip2 59 , and zstd 48 operate across diverse data types without domain-specific assumptions. Gzip integrates LZ77 76 and Huffman coding 34 , bzip2 builds on the Burrows-Wheeler transform 45 and move-to-front encoding 4 , while zstd uses fast LZ-style matching along with finite-state entropy coding 30 . Modality-specific compressors, in contrast, leverage domain-specific statistical structures for higher efficiency. For images, codecs like PNG 11 , WebP 29 , FLIF 62 , JPEG XL 63 , JPEG 2000 37 , and BPG 7 exploit spatial redundancy with filtering, prediction, or transform coding. For lossless speech compression, FLAC 18 uses linear predictive coding to model waveforms and compresses the residual signals with entropy coding. Learning-based Lossless Compressors. Learning-based compression has progressed rapidly in both lossy 14 ; 66 ; 27 ; 40 ; 41 ; 25 and lossless domains 5 ; 69 ; 46 ; 71 ; 67 ; 42 ; 22 ; 32 . We focus on lossless compression, which combines probability prediction with entropy coding to achieve bit-exact reconstruction. These methods are typically designed for a single modality, as the statistical properties of images, text, and speech differ greatly, making cross-modality generalization challenging. Existing text compressors use auto-regressive architectures to model token sequences 49 ; 64 ; 8 ; 38 ; 71 ; 9 , while image compressors attempt to capture the spatial dependencies 24 ; 13 ; 46 ; 5 ; 73 ; 72 ; 47 . While 22 ; 32 explore multi-modal lossless compression, they treat all modalities as byte or ASCII text, overlooking the modality heterogeneity and leading to sub-optimal performance. In addition, learning-based compressors, especially those built upon LLMs 24 ; 13 ; 49 ; 64 ; 9 ; 22 , often suffer from heavy computational complexity. 3 Proposed Method As outlined in Fig. 2 , OmniZip enables lossless compression for multiple modalities, supporting image-like data (natural image, medical image, tactile signal), text-like data (natural language, gene sequence, database), and speech. Input data is first converted into tokens \{x_{1},x_{2},...,x_{n}\} through a modality-unified tokenizer. A predictive model estimates each token’s contextual probability p(x_{i}|x_{ , < medical > , < tactile > , < text > , < gene > , < database > , or < speech > . Before softmax and arithmetic coding, modality masking is applied to zero out probabilities of non-target modalities, thereby reducing estimation errors and enhancing compression efficiency. For instance, when compressing images, only image-related token probabilities are retained. 3.3 Modality-Routing Context Learning In our RWKV backbone, each block consists of two components: a Time Mixing module 54 and a multilayer perceptron (MLP) 57 . The Time Mixing module models contextual dependencies through operations like token shift, R/K/V projection, WKV calculation, and state updates. As shown in Fig. 2 , different modalities exhibit distinct contextual dependencies. To enhance multi-modal processing, we integrate a mixture-of-experts (MoE) mechanism 12 into the Time Mixing module. Considering computational efficiency, we experiment with applying MoE to different components and find that applying MoE to the V layer yields impressive results. This phenomenon may stem from the inherent mechanism that, while the K layer serves as a semantic index and the R layer functions as a memory gating, the V layer represents the specific memory content. The diversity within the V layer is more crucial in handling multi-modal data. Therefore, we apply MoE only to the V layer, with the K and R layers shared across all V experts. Specifically, as shown in Fig. 3 , a learnable router assigns each token x_{i} a score g_{i,e} for each expert e : \small g_{i,e}=\text{softmax}(x_{i}W_{g})_{e}=\frac{\exp(x_{i}W_{g,e})}{\sum_{e^{\prime}=1}^{E}\exp(x_{i}W_{g,e^{\prime}})},\vskip-6.0pt (1) where W_{g} is the routing projection and E is the number of experts. Experts with the top- k scores then process the token. Herein, each expert is a V projection layer, and the router is a small feedforward network. The final output is the weighted sum of the selected experts’ outputs: \small\text{V}(x_{i})=\sum_{e\in\text{top}-k}\hat{g}_{i,e}\cdot e(x_{i})\vskip-7.0pt (2) where e(x_{i}) denotes the output of expert e for token x_{i} , and \hat{g}{i,e} is the re-normalized score. To ensure a compact model design, we experimentally use 4 experts, with top- k set to 2. This only adds the computational and parameter overhead of 3 additional V layers per Time Mixing module, which is negligible compared to the entire model. Figure 3 : One block of OmniZip’s predictive model. The model stacks N such blocks in total. Built on a lightweight RWKV7 backbone, it incorporates two modality-routing MoE modules for contextual learning and feedforward processing. 3.4 Modality-Routing Feedforward Process In RWKV, each block integrates a feedforward MLP to enhance the nonlinear representations 54 . However, using a general-purpose MLP is not well-suited for multi-modal processing, as it lacks the flexibility to effectively capture the distinct characteristics of different modalities. To better support multi-modal compression, we replace it with an MoE-based modality-routing feedforward module, as shown in Fig. 3 . The routing strategy follows Section 3.3 , except that the experts here are small MLPs instead of V projection layers. Each MLP expert uses a hidden factor of 2\times , half that of the original large MLP. With four experts and top- k=2 , the number of activated parameters per token is roughly the same as the original MLP, while the MoE design offers greater flexibility for multi-modal processing. 3.5 Loss Function Our loss function consists of two parts: (1) a primary cross-entropy loss that measures the divergence between the predicted and target distributions, and (2) auxiliary regularization terms that stabilize mixture-of-experts training, including the router Z-loss 77 and the load-balancing loss 61 . \displaystyle\mathcal{L} \displaystyle=\underbrace{-\sum q\log p}_{\text{Cross Entropy}} (3) \displaystyle+\lambda\underbrace{\frac{1}{T}\sum_{j=1}^{T}\left(\log\sum_{t=1}^{N}e^{x_{i,t}}\right)^{2}}_{\text{Z-loss}} \displaystyle+\mu\Big[\underbrace{\mathrm{CV}^{2}\!\left(\sum g_{i,e}\right)}_{\text{Expert Importance}}+\underbrace{\mathrm{CV}^{2}\!\left(\sum\mathbb{I}(g_{i,e}>0)\right)}_{\text{Load Balance}}\Big] As shown in Eq. 3 , the Z-loss penalizes large logits in the gating network, preventing instability during routing and accelerating convergence. The auxiliary loss encourages balanced expert utilization by penalizing the variance in expert importance and token assignment, where \mathrm{CV}^{2} denotes the squared coefficient of variation 1 , and \mathbb{I}(g_{i,e}>0) is the indicator of expert selection. The loss terms are weighted using \lambda=0.001 and \mu=0.01 . 4 Experiments 4.1 Experimental Setup Implementation Details. We vary the model size by adjusting the embedding dimension and the number of blocks, creating three model variants: OmniZip-S , OmniZip-M , and OmniZip-L , as shown in Table 2 . Table 2 : Details of the proposed OmniZip models. Model Blocks Embed Dim #Params MACs OmniZip-S 2 320 4.8M 3.88M OmniZip-M 2 912 38M 18.2M OmniZip-L 3 1488 152M 56.0M The models are trained using the FusedAdam optimizer 50 with a three-stage training strategy to stabilize the optimization of mixture-of-experts: (1) freeze the feedforward routing module and train the remaining parameters for two epochs with a learning rate of 1\times 10^{-4} ; (2) freeze the contextual routing module and train the other components for another two epochs with a learning rate of 1\times 10^{-4} ; (3) unfreeze all parameters and train the entire model for 20 epochs using a cosine annealing learning rate scheduler 43 , starting at 1\times 10^{-4} and decaying to 1\times 10^{-5} . All training process is conducted on NVIDIA A100 GPUs. Table 3 : Lossless compression results on image-like (natrual image, medical image, and tactile) datasets. Multi-modal compressors (except ours) are shown in gray. * denotes pretrained LLMs, {\dagger} indicates that some results are reprodeced by us. \prime indicates compression speeds measured on a MacBook CPU, and {\ddagger} denotes speeds on an A100 GPU (batch size 128). Other unmarked values are taken from their papers. Compressor #Params \downarrow MACs \downarrow Speed (KB/s) \uparrow Multi Modal bits/Byte \downarrow Kodak CLIC-P CLIC-M DIV2K TouchandGo ObjectFolder Axial Coronal Sagittal Classical gzip † 52 - - 14500 ′ ✓ 4.349 {}_{\textbf{(0\%)}} 4.039 {}_{\textbf{(0\%)}} 3.947 {}_{\textbf{(0\%)}} 4.224 {}_{\textbf{(0\%)}} 2.298 {}_{\textbf{(0\%)}} 3.969 {}_{\textbf{(0\%)}} 5.346 {}_{\textbf{(0\%)}} 4.563 {}_{\textbf{(0\%)}} 5.571 {}_{\textbf{(0\%)}} bzip2 † 59 - - 3300 ′ ✓ 4.359 4.033 3.931 4.208 2.288 4.031 5.403 4.617 5.628 zstd † 48 - - 11000 ′ ✓ 4.350 4.042 3.952 4.229 2.263 3.966 5.343 4.560 5.568 PNG † 11 - - 200 ′ ✗ 4.349 4.041 3.952 4.229 2.500 3.964 5.361 4.581 5.580 FLIF † 62 - - 652 ′ ✗ 2.903 2.792 2.497 2.914 0.808 3.765 4.860 3.999 5.160 BPG † 7 - - 180 ′ ✗ 3.382 3.144 2.855 3.285 0.936 3.726 5.004 4.167 5.337 WebP † 29 - - 330 ′ ✗ 3.205 3.006 2.774 3.176 0.739 3.612 4.944 4.161 5.214 JPEG-XL † 63 - - 970 ′ ✗ 2.902 2.722 2.395 2.826 0.739 3.657 4.719 3.891 5.091 JPEG2000 † 37 - - 5000 ′ ✗ 3.301 2.927 2.667 3.127 1.552 3.989 5.019 4.170 5.310 Learning-based \text{Llama3}^{*{\dagger}} 22 8B 7.80G 20 ‡ ✓ 4.862 4.290 4.234 4.378 2.455 4.060 5.514 4.832 5.784 \text{RWKV}^{*{\dagger}} 22 7B 7.19G 86 ‡ ✓ 4.937 4.310 4.298 4.718 2.523 4.118 5.244 4.712 5.529 DLPR ∗† 5 22.3M - 640 ✗ 2.860 2.380 2.160 2.550 1.082 3.774 - - - L3C ∗† 46 5M 6.86M 178 ✗ 3.260 2.940 2.640 3.090 1.350 3.842 5.160 4.450 5.520 RC 47 - - - ✗ - 2.930 2.540 3.080 - - - - - P2LLM 13 8B - 20 ‡ ✗ 2.830 2.350 2.080 2.510 - - - - - aiWave 67 695M - 40 ✗ - - - - - - 4.550 3.800 4.830 BCM 42 12.5M - - ✗ - - - - - - 4.410 3.630 4.810 Ours OmniZip-S 4.8M 3.88M 1223 ‡ ✓ 3.307 {}_{\textbf{(-24\%)}} 2.933 {}_{\textbf{(-27\%)}} 2.620 {}_{\textbf{(-34\%)}} 2.968 {}_{\textbf{(-30\%)}} 1.338 {}_{\textbf{(-42\%)}} 3.275 {}_{\textbf{(-18\%)}} 4.560 {}_{\textbf{(-15\%)}} 4.179 {}_{\textbf{(-8\%)}} 5.286 {}_{\textbf{(-5\%)}} OmniZip-M 38M 18.2M 1015 ‡ ✓ 3.103 {}_{\textbf{(-29\%)}} 2.926 {}_{\textbf{(-28\%)}} 2.378 {}_{\textbf{(-40\%)}} 2.749 {}_{\textbf{(-35\%)}} 1.110 {}_{\textbf{(-52\%)}} 2.973 {}_{\textbf{(-25\%)}} 4.531 {}_{\textbf{(-15\%)}} 3.880 {}_{\textbf{(-15\%)}} 4.969 {}_{\textbf{(-11\%)}} OmniZip-L 152M 56.0M 420 ‡ ✓ 2.925 {}_{\textbf{(-33\%)}} 2.578 {}_{\textbf{(-36\%)}} 2.273 {}_{\textbf{(-42\%)}} 2.542 {}_{\textbf{(-40\%)}} 0.987 {}_{\textbf{(-57\%)}} 2.689 {}_{\textbf{(-32\%)}} 4.466 {}_{\textbf{(-16\%)}} 3.837 {}_{\textbf{(-16\%)}} 4.848 {}_{\textbf{(-13\%)}} Table 4 : Lossless compression results on text-like (natural text, gene sequence, database) and speech datasets. Multi-modal compressors (except ours) are shown in gray. * denotes pretrained LLMs, {\dagger} indicates that some results are reprodeced by us. \prime indicates compression speeds measured on a MacBook CPU, and {\ddagger} denotes speeds on an A100 GPU (batch size 128). Other values are taken from their papers. Compressor #Params \downarrow MACs \downarrow Speed (KB/s) \uparrow Multi Modal bits/Byte \downarrow enwik9 Gutenberg Spider WikiSQL GenoSeq DNACorpus LibriSpeech Classical gzip † 52 - - 14500 ′ ✓ 2.590 {}_{\textbf{(0\%)}} 3.819 {}_{\textbf{(0\%)}} 2.289 {}_{\textbf{(0\%)}} 1.672 {}_{\textbf{(0\%)}} 2.390 {}_{\textbf{(0\%)}} 2.318 {}_{\textbf{(0\%)}} 6.511 {}_{\textbf{(0\%)}} bzip2 † 59 - - 3300 ′ ✓ 2.082 3.607 2.326 1.403 2.202 2.148 5.647 zstd † 48 - - 11000 ′ ✓ 2.364 3.703 2.258 1.520 2.404 2.324 6.467 \text{FLAC}^{{\dagger}} 18 - - 2000 ′ ✗ - - - - - - 4.961 Learning-based \text{Llama3}^{*{\dagger}} 22 8B 7.80G 20 ‡ ✓ 0.722 1.030 0.770 0.645 1.889 1.808 3.616 \text{RWKV}^{*{\dagger}} 22 7B 7.19G 86 ‡ ✓ 0.774 1.036 0.723 0.669 1.900 1.890 3.589 \text{tszip}^{{\dagger}} 9 169M 131M 180 ✗ 1.083 1.169 0.768 0.967 1.969 1.965 - \text{NNCP}^{{\dagger}} 8 - 187M 1.6 ✗ 0.853 1.320 1.033 0.700 1.732 1.748 - \text{CMIX}^{{\dagger}} 38 - - 4 ✗ 0.879 1.363 1.056 0.629 1.656 1.667 - \text{L3TC}^{{\dagger}} 71 12M 13.0M 580 ✗ 1.281 2.237 2.270 1.712 2.368 2.346 - Ours OmniZip-S 4.8M 3.88M 1223 ‡ ✓ 1.370 {}_{\textbf{(-47\%)}} 1.552 {}_{\textbf{(-60\%)}} 1.320 {}_{\textbf{(-42\%)}} 1.170 {}_{\textbf{(-30\%)}} 1.868 {}_{\textbf{(-24\%)}} 2.015 {}_{\textbf{(-22\%)}} 4.155 {}_{\textbf{(-36\%)}} OmniZip-M 38M 18.2M 1015 ‡ ✓ 1.009 {}_{\textbf{(-61\%)}} 1.278 {}_{\textbf{(-67\%)}} 1.298 {}_{\textbf{(-43\%)}} 0.842 {}_{\textbf{(-50\%)}} 1.777 {}_{\textbf{(-26\%)}} 1.810 {}_{\textbf{(-22\%)}} 3.852 {}_{\textbf{(-41\%)}} OmniZip-L 152M 56.0M 420 ‡ ✓ 0.980 {}_{\textbf{(-62\%)}} 1.175 {}_{\textbf{(-69\%)}} 1.243 {}_{\textbf{(-46\%)}} 0.787 {}_{\textbf{(-53\%)}} 1.740 {}_{\textbf{(-27\%)}} 1.782 {}_{\textbf{(-23\%)}} 3.810 {}_{\textbf{(-42\%)}} Multi-Modality Datasets. Our dataset covers seven types: natural image, medical image, tactile signal, text, gene sequence, database, and speech. In total, 16 datasets are utilized for training and evaluation: • Image: we train on the training set of DIV2K 2 , and evaluate on Kodak 21 , CLIC-Pro, CLIC-Mobile 17 , and the test set of DIV2K. • Medical: we adopt MRNet Axial, Coronal, and Sagittal datasets 10 and follow their official train–test splits. • Text: the training set consists of enwik8 19 and 2000 Gutenberg eBooks 56 . The test set is enwik9 20 and additional 1000 Gutenberg eBooks. • Speech: we use the LibriSpeech dataset 51 with its official training and test splits. • Database: we train on the Spider training set 70 , and evaluate on both WikiSQL 35 and the Spider test set. • Gene: we use GenoSeq 16 and DNACorpus 55 datasets, and follow their official train-test splits. • Tactile: the training and test sets follow the official partitions of TouchandGo 68 and ObjectFolder 28 . To ensure balanced training across modalities, we control each modality’s training set to 1GB via random sampling and data augmentation. Further, we also implement a balanced batch sampler to ensure even sampling from each modality in every batch by cycling through shuffled indices. Compared Methods and Metrics. We compare lossless compression across different modalities. Classical baselines include general-purpose lossless compressors (gzip 52 , bzip2 59 , zstd 48 ), image-specific compressors (PNG 11 , WebP 29 , FLIF 62 , JPEG XL 63 , JPEG2000 37 , BPG 7 ), and the speech lossless compressor FLAC 18 . For learning-based methods, we evaluate recent image lossless compressors 5 ; 13 ; 46 ; 47 ; 67 ; 42 , text compressors 8 ; 38 ; 71 ; 9 , and multi-modal compressors 22 which compresses data using pretrained LLMs 3 ; 53 . Herein, image-like data (natural images, medical images, and tactile data) can be compressed using image and general-purpose compressors, while text-like data (natural language, gene, and databases) are compressed using text and general-purpose compressors. There is currently no dedicated learning-based lossless speech compressor. We evaluate compression performance using bits/Byte as metrics. Model complexity is assessed by MACs and inference speed (KB/s) on multiple devices (an NVIDIA A100 GPU, a MacBook Pro CPU, and an iPhone 17 Pro NPU). 4.2 Multi-Modality Lossles Results Table 3 and Table 4 compare OmniZip with existing lossless compression methods. The former reports results on image-like data (natural image, medical image, and tactile data), while the latter presents results on text-like data (natural text, gene sequence, database) and speech. Both tables include general-purpose multi-modal compressors (colored in gray) and modality-specific compressors for comparison. The radar chart in Fig. 1 compares multi-modal compressors across different modalities, while Fig. 4 offers comparisons of learning-based compressors on individual datasets. Compression Results on Image-like Data. As shown in Table 3 , general-purpose classical compressors such as gzip, bzip2, and zstd show limited effectiveness on image-like data, achieving about 4.3, 2.3, 4.6 bits/Byte on Kodak, TouchandGo, and Coronal, respectively. Among classical image compressors, FLIF and JPEG-XL perform the best, offering roughly 1.5\times \sim 2.5\times better compression than gzip. Learning-based methods exhibit clear domain dependency. DLPR, L3C, RC, and P2LLM, all trained on natural images, perform worse on tactile and medical data. Though 22 presents as a general-purpose compressor, it compresses images by converting them into ASCII sequences and applying pretrained LLMs 3 ; 53 for probability prediction, resulting in sub-optimal performance. In contrast, OmniZip achieves pleasing compression across all image-like modalities. Even the smallest version, OmniZip-S, performs comparably to specialized learning-based methods and outperforms gzip by about 30% on medical, tactile, and natural images. OmniZip-M and OmniZip-L further improve compression performance close to state-of-the-art levels, reaching 2.925, 0.987, and 3.937 bits/Byte on Kodak, TouchandGo, and Coronal datasets, respectively. Compression Results on Text-like Data. Table 4 presents the lossless compression results on natural language, gene sequence, and database. Classical compressors like gzip, bzip2, and zstd achieve moderate performance (about 2.3 bits/Byte on enwik9, Spider, and GenoSeq). Learning-based text compressors perform strongly on natural language data, with NNCP and CMIX reaching 0.85 and 0.88 bits/Byte on enwik9. Pretrained LLMs compress natural language more effectively, but are computationally heavy and yield only modest gains on gene sequences. Figure 4 : Comparison of learning-based lossless compressors across multi-modal datasets. The x-axis shows the model size (in millions of parameters), and the y-axis indicates compression efficiency (bits/Byte, lower is better). Models closer to the lower-left corner achieve better compression with fewer parameters. The dashed orange line represents the performance baseline of gzip. OmniZip achieves competitive compression performance on these text-like datasets with much fewer parameters. OmniZip-S achieves about 24% to 47% performance gains compared to gzip across all text-like datasets. OmniZip-M and OmniZip-L, further enhance compression performance by another 10% \sim 20%. The performance gains on database and gene data are more pronounced, with OmniZip-L achieving near SOTA compression (1.740 and 1.782 bits/Byte) on GenoSeq and DNACorpus datasets. Compression Results on Speech Data. As shown in Table 4 , classical compressors such as gzip, bzip2, and zstd achieve 5.6 \sim 6.5 bits/Byte on LibriSpeech, while the specialized speech compressor FLAC performs better at 4.96 bits/Byte. Currently, there is no dedicated learning-based lossless speech compressor. The multi-modal method 22 provides promising compression (about 3.6 bits/Byte on LibriSpeech) but suffers from heavy complexity. In contrast, OmniZip offers a superior balance between compression and efficiency, achieving 36% \sim 42% improvements over gzip and 15 \sim 23% over FLAC on LibriSpeech. Figure 5 : OmniZip’s inference speed across various platforms (CPU of MacBook Pro, NPU of iPhone17 Pro, and GPU of NVIDIA A100) and batch sizes (1, 16, 128, 512, 1024). 4.3 Parameter and Complexity Analysis We evaluate OmniZip’s inference speed across different hardware platforms and batch sizes, as shown in Fig. 5 . The inference speed improves as the batch size increases and saturates around a batch size of 512. Leveraging the modality-routing sparse design, OmniZip activates only a subset of parameters per token and achieves high computational efficiency. Specifically, on an NVIDIA A100 GPU, all three models achieve MB/s-level inference speeds, with OmniZip-S peaking at approximately 4MB/s. On resource-constrained devices like MacBook CPU and iPhone NPU, OmniComp-S achieves a near real-time speed of \sim 1MB/s, and the larger variants reach up to \sim 200KB/s. Besides, we take OmniZip-S as an example to illustrate the routing patterns across modalities and modules, as shown in Fig. 6 . Each bar represents the percent of expert activations for a given modality within a routing module (context learning or feedforward). We observe that context-learning routing exhibits more uneven expert utilization, where some experts may dominate specific modalities. In contrast, feedforward routing shows a more balanced expert usage, suggesting that contextual routing may capture more modality-specific dynamics, whereas feedforward routing tends to promote more generalized representation learning. Figure 6 : Expert usage of OmniZip-S across blocks and routing modules. Each bar shows the expert usage percent (%) for a modality in a routing module (context learning or feedforward). 4.4 Ablation Studies Table 5 : Ablations on all our proposals: modality-unified tokenization, modality-routing contextual learning, modality-routing feedforward, and reparameterization training. Compression performance (bits/Byte) is evaluated on representative dataset from each modality, and the inference speed is measured on a MacBook CPU with a batch size of 128. The chosen configuration is colored in orange. Unifed Tokenizer Contextual Routing Feedforward Routing Reparam Multi Modal #Params \downarrow MACs \downarrow Speed (KB/s) \uparrow bits/Byte \downarrow Kodak TouchandGo Coronal enwik9 Spider GenoSeq LibriSpeech ✗ ✗ ✗ ✗ ✗ 3.2M 3.68M 856 - - - 1.658 - - - ✗ ✗ ✗ ✓ ✗ 3.2M 3.68M 856 - - - 1.590 - - - ✓ ✗ ✗ ✓ ✓ 3.2M 3.68M 856 3.383 1.453 4.545 1.660 1.662 1.984 4.365 ✓ ✗ ✓ ✓ ✓ 4.0M 3.73M 633 3.352 1.431 4.314 1.424 1.448 1.915 4.295 ✓ ✓ ✗ ✓ ✓ 3.0M 3.82M 780 3.339 1.419 4.188 1.410 1.482 1.889 4.302 ✓ ✓ ✓ ✗ ✓ 4.8M 3.88M 500 3.414 1.360 4.260 1.390 1.356 1.895 4.183 ✓ ✓ ✓ ✓ ✓ 4.8M 3.88M 500 3.307 1.338 4.179 1.370 1.320 1.868 4.155 Table 6 : Ablations on modality-routing contextual learning. Routing is applied to all blocks with 4 experts and top- k =2. We report the total parameters and parameters activated per token. Compression performance (bits/Byte) is evaluated on representative dataset from each modality, and the inference speed is measured on a MacBook CPU with a batch size of 128. The chosen configuration is colored in orange. Contextual Routing #Params \downarrow MACs \downarrow Speed (KB/s) \uparrow bits/Byte \downarrow Receptance Key Value Total Activated Kodak TouchandGo Coronal enwik9 Spider GenoSeq LibriSpeech ✓ ✗ ✗ 4.8M 2.7M 3.88M 500 3.315 1.372 4.224 1.311 1.324 1.795 4.281 ✗ ✓ ✗ 4.8M 2.7M 3.88M 500 3.614 1.533 4.398 1.388 1.470 1.862 4.391 ✗ ✗ ✓ 4.8M 2.7M 3.88M 500 3.307 1.338 4.179 1.370 1.320 1.868 4.155 ✓ ✓ ✗ 5.4M 3.0M 4.02M 436 3.317 1.352 4.080 1.292 1.361 1.860 4.280 ✓ ✗ ✓ 5.4M 3.0M 4.02M 436 3.313 1.360 4.140 1.297 1.309 1.813 4.178 ✗ ✓ ✓ 5.4M 3.0M 4.02M 436 3.302 1.361 4.188 1.290 1.467 1.835 4.218 ✓ ✓ ✓ 6.0M 3.3M 4.17M 382 3.358 1.408 4.122 1.265 1.481 1.830 4.291 Table 7 : Ablations on the configurations of the two modality-routing modules (e.g., routed blocks, expert count, top- k ). We report the total parameters and parameters activated per token. Compression performance (bits/Byte) is evaluated on representative dataset from each modality, and the inference speed is measured on a MacBook CPU with a batch size of 128. The chosen configuration is colored in orange. Blocks top- k Experts Hidden Factor #Params \downarrow MACs \downarrow Speed (KB/s) \uparrow bits/Byte \downarrow Total Activated Kodak TouchandGo Coronal enwik9 Spider GenoSeq LibriSpeech 1 1 4 2\times 3.6M 2.2M 3.65M 594 3.436 1.150 4.267 1.391 1.511 1.878 4.336 1 2 4 2\times 3.6M 2.5M 3.85M 570 3.358 1.420 4.230 1.380 1.344 1.896 4.307 2 2 4 2\times 4.8M 2.7M 3.88M 500 3.307 1.338 4.179 1.370 1.320 1.868 4.155 2 2 3 2\times 4.0M 2.7M 3.88M 500 3.355 1.424 4.154 1.369 1.410 1.882 4.300 2 2 5 2\times 5.2M 2.7M 3.88M 500 3.232 1.416 4.190 1.358 1.352 1.855 4.206 2 2 4 1\times 3.2M 2.0M 3.48M 533 3.502 1.560 4.216 1.392 1.590 1.875 4.380 2 2 4 2\times 4.8M 2.7M 3.88M 500 3.307 1.338 4.179 1.370 1.320 1.868 4.155 2 2 4 4\times 8.0M 4.4M 4.65M 325 3.316 1.354 4.110 1.331 1.315 1.850 4.157 Discussion on Multi-Modal Proposals. Table 5 reports ablations of our multi-modal components: the unified tokenizer enables multi-modality, while contextual and feedforward routing provide modality adaptivity in context modeling and nonlinear transformation. We run all ablations on OmniZip-S with the same hyperparameters as the main experiments. It can be seen that removing either the routing mechanism leads to a clear drop in multi-modal compression performance, confirming their effectiveness in enhancing multi-modal compression. Discussion on Modality-Routing Configurations. We analyze the configurations of the two modality-routing modules in contextual learning and feedforward stages. For contextual learning, we apply routing to different projection layers with 4 experts and top- k of 2. As shown in Table 6 , applying routing to the V projection yields better compression performance than applying it to the K/R projections. Extending routing to more projections slightly increases model capacity but also leads to 10% \sim 20% more parameters. Hence, we apply routing solely to the V projection in each contextual learning stage. Then, we vary the number of routed blocks (from half to all), expert count (from 3 to 5), and top- k (from 1 to 2) for the contextual and feedforward routing modules, as shown in Table 7 . It can be seen that using 4 experts and top- k of 2 yields the best trade-off between complexity and performance. We also vary the hidden factor of the feedforward routing module’s MLP experts. Moreover, setting the feedforward experts with a hidden factor of 2\times strikes a good balance between compression performance and model compactness, hence it is adopted for OmniZip models. 5 Conclusion We present OmniZip, a unified and lightweight lossless compressor designed for multi-modal data. It supports most real-world data types suitable for lossless compression: image-like data (e.g., natural image, medical image, tactile signals), text-like data (e.g., natural language, gene sequence, database), and speech. Built on a lightweight backbone, OmniZip incorporates three components to enable efficient multi-modal compression: modality-unified tokenization, modality-routing context learning, and modality-routing feedforward. A reparameterization training strategy is also used to improve model capacity. OmniZip achieves compression performance on par with or better than existing state-of-the-art methods across multiple modalities, while using much fewer parameters. It also supports near real-time inference even on resource-constrained edge devices, like MacBook CPUs and iPhone NPUs. 6