ZipLLM : Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression
Zirui Wang, Tingfeng Lan, Zhaoyuan Su, Juncheng Yang, Yue Cheng · 2025
Read ZipLLM : Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression by Zirui Wang, Tingfeng Lan, Zhaoyuan Su, Juncheng Yang, Yue Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06252.txt.
Introduction
Zirui Wang and co-authors shrink storage for large language models on public hubs by over half, in work from twenty twenty-five. Hubs now hold tens of petabytes of models and variants. Fine-tuned variants far outnumber base models and eat most storage. Classic dedup and compression ignore model structure or clash with each other. A sweep of all public hub models reveals exploitable patterns.
Tuned kin differ from base models in sparse, structured ways. Bitwise likeness can cluster models into families reliably. Tensor-level dedup fits model workloads with low overhead costs. A new lossless method squeezes the difference between tuned and base weights. The full pipeline, ZipLLM, unifies dedup with that delta squeeze. Storage falls by more than half, beating prior best methods clearly. Hubs can serve more models from the same disk budget.
Catalog record
| CREATOR | Zirui Wang, Tingfeng Lan, Zhaoyuan Su, Juncheng Yang, Yue Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-06252
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ZipLLM : Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression Zirui Wang, Tingfeng Lan, Zhaoyuan Su, Juncheng Yang, Yue Cheng Abstract Modern model hubs, such as Hugging Face, store tens of petabytes of LLMs, with fine-tuned variants vastly outnumbering base models and dominating storage consumption. Existing storage reduction techniques—such as deduplication and compression—are either LLM-oblivious or not compatible with each other, limiting data reduction effectiveness. Our large-scale characterization study across all publicly available Hugging Face LLM repositories reveals several key insights: (1) fine-tuned models within the same family exhibit highly structured, sparse parameter differences suitable for delta compression; (2) bitwise similarity enables LLM family clustering; and (3) tensor-level deduplication is better aligned with model storage workloads, achieving high data reduction with low metadata overhead. Building on these insights, we design BitX , an effective, fast, lossless delta compression algorithm that compresses the XORed difference between fine-tuned and base LLMs. We build ZipLLM , a model storage reduction pipeline that unifies tensor-level deduplication and lossless BitX compression. By synergizing deduplication and compression around LLM family clustering, ZipLLM reduces model storage consumption by 54%, over 20% higher than state-of-the-art deduplication and compression approaches. 1 Introduction Large language models (LLMs) have become foundational tools in modern artificial intelligence (AI). With the rapid progress in open-source LLM development [ 44 , 47 , 46 , 34 , 48 ] , millions of LLMs are now publicly available through model hubs such as Hugging Face [ 18 ] and TensorFlow Hub [ 23 ] . These platforms support uploads, downloads, and sharing of base models and fine-tuned variants, enabling users to adapt models to diverse downstream tasks with minimal effort. This trend has led to an explosion in the number of hosted models. As shown in Figure 1 , Hugging Face alone hosts over 14 petabytes (PB) of models (as of Q1 2025), with storage volume growing exponentially, posing a serious threat to the sustainability of machine learning (ML) infrastructure. Table 1 : Comparison of model storage reduction techniques. Note that existing solutions are limited to use either deduplication or compression. Solution Compression Deduplication Cross-model Throughput Storage Reduction Cons & Pros HuggingFace Xet [ 79 ] No Yes Yes Low High No compression support ELF [ 70 ] Yes No No High High Lossy compression ZipNN [ 28 ] Yes No No Medium Medium Ignores cross-model redundancy FM-Delta [ 58 ] Yes No Yes Low Medium Requires identical model structure; lacks BF16 support ZipLLM (ours) Yes Yes Yes High High Lossless and model structure-aware dedup and compression Two observations underscore this challenge. First, fine-tuned LLMs vastly outnumber base models and contribute disproportionately to overall storage footprint, despite being only slight modifications. Second, LLM storage is dominated by two floating-point formats: BF16 and FP32 . While FP32 is popular in terms of model count (often in smaller models such as those for computer vision), BF16 accounts for the majority of total LLM storage size. These trends highlight the need to prioritize LLM-specific storage patterns in future optimizations. Figure 1 : Left: Hugging Face’s model count and storage consumption grow exponentially. Right: ZipLLM achieves both high data reduction and throughput. ZipLLM represents the end-to-end system throughput. while BitX shows the compression kernel throughput. We collect all public LLM repositories from Hugging Face (Cutoff date March 2025) and conduct a first-of-its-kind, large-scale study focusing on LLM storage . Our analysis leads to the following insights : • Element-wise weight deltas are small and structured within LLM model families. Fine-tuned models derived from the same base exhibit tiny differences, making them ideal for lossless delta compression. • Bitwise similarity enables LLM clustering and lineage tracking. Bit distance, a new metric that we propose, based on the bitwise Hamming distance, serves as a lightweight, robust signal for identifying LLM families and potentially supporting applications like model provenance, duplicate detection, and clustering. • Chunk-based deduplication is LLM-oblivious and suboptimal for modern model storage. Chunk-level deduplication, such as content-defined chunking (CDC) [ 53 , 82 , 56 ] , operates on raw byte streams without LLM structure awareness, resulting in the loss of crucial information needed for effective model-aware compression. It also scales poorly with storage capacity. • Model-aware, tensor-level deduplication is well-suited for LLM-aware lossless compressors , offering reasonable data reduction, but with significantly higher performance and lower metadata overhead compared to CDC. This paper makes the following contributions: • We conduct a comprehensive analysis of Hugging Face’s massive-scale model repository, with a focus on how LLM family structure impacts storage redundancy and compression effectiveness. • We introduce a novel metric, bit distance , to quantify the similarity between fine-tuned models and their base models. • Building on this, we design BitX , a highly effective, fast, lossless delta compression algorithm that compresses LLM variants by encoding XOR-based deltas. • We identify a new ML system design principle : for modern model storage systems, deduplication and lossless compression must be co-designed and unified to fully exploit model structure and redundancy. • We build ZipLLM , a model storage reduction pipeline that synergizes tensor-level deduplication and lossless BitX compression, achieving higher storage savings for large-scale LLM repositories. Evaluation results show that ZipLLM reduces the storage size of 3,048 sampled LLMs by 54.1%, 20% higher than the state-of-the-art methods. Meanwhile, ZipLLM achieves 2 \times higher compression throughput (Figure 1 ). Our implementation is publicly available at: https://github.com/ds2-lab/ZipLLM . 2 Background and Related Work This section reviews existing techniques for reducing data storage, including both general-purpose and model-aware approaches. A high-level comparison of model storage compression and deduplication methods is summarized in Table 1 . 2.1 Traditional Storage Reduction General-purpose Compression. General-purpose data compression techniques, such as Zstandard (zstd) [ 8 ] and Brotli [ 1 ] , are widely used in storage systems to reduce data size by exploiting local byte-level redundancy through methods like dictionary-based compression [ 91 , 77 ] and entropy coding [ 74 , 30 , 13 , 17 , 43 , 62 ] . While these methods are data-type agnostic, further compression gains can be achieved when the data type is known. For instance, run-length encoding and delta encoding are highly effective for low-entropy [ 69 ] and file changes [ 73 , 42 ] , and specialized techniques have been developed for columnar and time-series datasets [ 35 , 75 , 36 , 5 , 40 , 7 ] . Lossy compression methods—such as ZFP [ 15 , 39 ] and SZ [ 37 , 14 , 71 ] —though effective in scientific domains, are unsuitable for model storage due to their inability to guarantee exact recovery. Similarly, while quantization is a popular lossy compression approach for model inference [ 38 , 19 , 83 , 88 , 87 , 12 ] , it is a user-driven choice and orthogonal to storage system design. For model storage, lossless compression that preserves floating-point precision remains essential for correctness. General-purpose Deduplication. Deduplication is a widely used technique for reducing storage footprint by identifying and storing only unique data blocks, replacing duplicates with references. File-level deduplication, used in systems like Git LFS [ 22 ] , eliminates exact file copies with minimal metadata overhead but cannot detect partial