Dynamic Base Model Shift for Delta Compression
Chenyu Huang, Peng Ye, Shenghe Zheng, Xiaohui Wang, Lei Bai, Tao Chen, Wanli Ouyang · 2025
Read Dynamic Base Model Shift for Delta Compression by Chenyu Huang, Peng Ye, Shenghe Zheng, Xiaohui Wang, Lei Bai, Tao Chen, Wanli Ouyang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11344.txt.
Introduction
Every fine-tuned copy of a big model costs storage, and compressing only the changes helps, until the compression bites. Chenyu Huang and co-authors question the fixed reference point those deltas are measured from in a 2025 paper.
Standard practice measures each fine-tune against the original pre-trained model, then prunes or rounds the difference. The team shows that starting point is rarely the best one. Their method shifts the base model dynamically for each task, so the remaining delta shrinks and survives heavier compression. That matters most at extreme compression rates, where the fixed-base approach degrades sharply. Cheaper deltas mean one shared base can serve many tasks on modest hardware.
Catalog record
| CREATOR | Chenyu Huang, Peng Ye, Shenghe Zheng, Xiaohui Wang, Lei Bai, Tao Chen, Wanli Ouyang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamic Base Model Shift for Delta Compression Chenyu Huang, Peng Ye, Shenghe Zheng, Xiaohui Wang, Lei Bai, Tao Chen, Wanli Ouyang Abstract Transformer-based models with the pretrain-finetune paradigm bring about significant progress, along with the heavy storage and deployment costs of finetuned models on multiple tasks. Delta compression attempts to lower the costs by reducing the redundancy of delta parameters (i.e., the difference between the finetuned and pre-trained model weights) through pruning or quantization. However, existing methods by default employ the pretrained model as the base model and compress the delta parameters for every task, which may causes significant performance degradation, especially when the compression rate is extremely high. To tackle this issue, we investigate the impact of different base models on the performance of delta compression and find that the pre-trained base model can hardly be optimal. To this end, we propose Dynamic Base Model Shift ( DBMS ), which dynamically adapts the base model to the target task before performing delta compression. Specifically, we adjust two parameters, which respectively determine the magnitude of the base model shift and the overall scale of delta compression, to boost the compression performance on each task. Through the low-cost learning of these two parameters, our DBMS can maintain most of the finetuned model’s performance even under an extremely high compression ratio setting, significantly surpassing existing methods. Moreover, our DBMS is orthogonal and can be integrated with a variety of other methods. It has been evaluated across different types of models, including language models, vision transformers, and multi-modal models. 1 Introduction As transformer-based models become mainstream vaswani2017attention ; devlin2018bert ; achiam2023gpt ; liu2024deepseek along with the pretrain-finetune paradigm, the model size and number are becoming increasingly larger, thus storing models finetuned on different downstream tasks is getting much more expensive. To this end, different techniques have been proposed to reduce storage and deployment costs. A representative technique is model merging, which means fusing model weights from different models wortsman2022model ; RegMean ; TIESMerging . Though showing impressive performance and wide applicability yang2024model , model merging still suffers from significant performance degradation when the number or difficulty of target tasks increases huang2024emrmerging ; lu2024twin ; zheng2024free ; tang2024merging ; tang2025data ; kim2025task . Therefore, delta compression, which compresses the delta parameters or task vectors Task_Arithmetic , i.e., the difference between the finetuned and the pre-trained model, has been recently proposed liu2024bitdelta ; ping2024deltacome . Similar to model merging, delta compression also utilizes the extremely redundant properties of delta parameters of finetuned models, but differently, delta compression is not affected by weight interference, thus achieving more stable performance. Currently, delta compression methods can be roughly divided into pruning-based methods and quantization-based methods. Pruning-based methods eliminate most elements within delta parameters randomly yu2023language ; deng2024dare or based on magnitude TIESMerging ; deng2024dare . Quantization-based methods apply post-training quantization (PTQ) frantar2022gptq to delta parameters to quantize them to 1-bit liu2024bitdelta or equivalently 1-bit ping2024deltacome ; huang2025seeing through mixed-precision quantization. Both pruning-based and quantization-based methods can compress the delta parameters, thus significantly reducing the storage costs. However, existing methods apply the pre-trained model as the base model for delta compression by default, which may not be optimal. As shown in the left part of Fig. 1 , there is much room for lowering the distance between the base model and different finetuned models. In comparison, after applying our Dynamic Base Model Shift , the distance could be significantly reduced, as shown in Fig. 1 (c). To resolve the mentioned issue, we rethink and analyze the existing delta compression paradigm. When handling N downstream tasks \left[T_{1},T_{2},..,T_{N}\right] , which includes N model weights \left[\theta_{1},\theta_{2},..,\theta_{N}\right] finetuned from the same pre-trained model \theta_{pre} , the goal of existing methods is to compress the differences between finetuned models and the pre-trained model, which can be formulated as: \delta_{t}=\mathcal{C}\left(\theta_{t}-\theta_{base}\right), where \mathcal{C}\left(\cdot\right) denotes the delta compression method, t\in N , and \theta_{base} is the pretrained model weight \theta_{pre} . However, as illustrated before, the optimal base model for each task varies. A single base model may cause severe performance loss on partial tasks and thus cannot achieve optimal performance. Therefore, we propose a new paradigm. Instead of a single static base model, we apply a dynamic base model, which can be formulated as: \theta_{base}^{t}=\theta_{pre}+\lambda_{1}^{t}\cdot\tau_{base}, (1) where base vector \tau_{base}=\mathcal{C}\left(\frac{\sum_{t=1}^{N}\left(\theta_{t}-\theta_{pre}\right)}{N}\right) is the compressed delta parameter (by BitDelta liu2024bitdelta in our experiments) of the average model weights and \lambda_{1}^{t} is a task-specific modulator to adapt the base model to the target task, thus reducing the performance loss. Based on these findings, we propose Dynamic Base Model Shift ( DBMS ). We first initialize the shifted base model based on the minimizing L2 distance strategy. Then, to further boost delta compression’s performance, we apply another scaling coefficient \lambda_{2}^{t} on the compressed task-specific delta parameter, that is: Figure 1: Comparison between vanilla delta compression methods (left) and our DBMS (right), which first (a) obtains a compressed base vector, then we (b) fuse the base vector with the ajusted magnitude to the pretrained model to realize dynamic base model shift. Finally we (c) further dynamically adjust the scales of delta parameters to boost the performance. \delta_{t}^{\prime}=\lambda_{2}^{t}\cdot\mathcal{C}\left(\theta_{t}-\theta_{base}^{t}\right). (2) To find the optimal \lambda_{1}^{t} and \lambda_{2}^{t} values for task t , we apply low-cost training on the parameters \lambda_{1}^{t} and \lambda_{2}^{t} . During the training process, a partial (10% in our experiments) unlabeled test dataset is required. Through training, the performance of delta compression can be effectively improved at a low cost. The schematic diagram of our DBMS is shown in the right part of Fig. 1 . In this paper, we demonstrate the effectiveness of our DBMS under the setting of (1) Language models, including RoBERTa liu2019roberta , GPT-2 radford2019language , and Flan-T5 chung2024scaling ; raffel2020exploring models. The performance improvement is remarkable under all the compression rates and model structure settings, especially when the compression rate is extremely high. For example, our DBMS can improve the performance of GPT-2 models by over 15\% under the compression ratio of 99%. (2) Vision transformer dosovitskiy2020image (ViT) models, including ViT-B/32 and ViT-L/14 variants. DBMS can improve the performance of ViT-B/32 and ViT-L/14 models by over 70\% and 40\% under the compression ratio of 99.8\% . (3) Multi-modal models BEiT3 beit3 . DBMS shows effectiveness when applied to different kinds of multi-modal tasks. In short, the proposed DBMS can achieve impressive performance under all the experimental settings, especially under an extremely high compression ratio setting. More importantly, it can be combined with pruning- and quantization-based delta compression methods. Our contributions can be summarized as: (1) We find that the default paradigm (regarding the pre-trained model as the base model) of existing delta compression methods may not be optimal. We first investigate this problem and propose Dynamic Base Model Shift , dubbed DBMS , to shift the base model dynamically, thus solving the performance degradation problem. (2) We apply two parameters for each task to scale the compressed base vector and the delta parameter. By low-cost fine-tuning of these two parameters, the performance of delta compression can be significantly improved. (3) DBMS is orthogonal to the existing delta compression method and can be effectively combined with both pruning- and quantization-based methods to boost the performance at