Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transform

Chenyu Huang, Peng Ye, Xiaohui Wang, Shenghe Zheng, Biqing Qi, Lei Bai, Wanli Ouyang, Tao Chen · 2025

Read Seeing Delta Parameters as JPEG Images: Data-Free Delta Compression with Discrete Cosine Transform by Chenyu Huang, Peng Ye, Xiaohui Wang, Shenghe Zheng, Biqing Qi, Lei Bai, Wanli Ouyang, Tao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06676.txt.

Description

Chenyu Huang and co-authors present Delta-DCT in a 2025 paper on shrinking the extra weights of fine-tuned models. Modern practice pretrains one large model, then fine-tunes a copy per task, and storing every copy is costly. Delta compression trims the gap between each copy and the base, but older methods need training data or extra training.

The trick borrows from JPEG image compression. Weight gaps are cut into patches, each patch is scored for importance, and patches move into the discrete cosine transform domain for quantization at matched bit widths. No training or calibration data is needed at any step. Tests span large language models, smaller text models, vision transformers, and a multimodal model. Even at very tight compression, scores matched or beat the uncompressed fine-tuned models in many cases.

Catalog record

CREATORChenyu Huang, Peng Ye, Xiaohui Wang, Shenghe Zheng, Biqing Qi, Lei Bai, Wanli Ouyang, Tao Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06676

Browse the full library · About the Tower of Babel