CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging

Zongzhen Yang, Binhang Qi, Hailong Sun, Wenrui Long, Ruobing Zhao, Xiang Gao · 2025

Read CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging by Zongzhen Yang, Binhang Qi, Hailong Sun, Wenrui Long, Ruobing Zhao, Xiang Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01874.txt.

Description

Zongzhen Yang and co-authors improve in 2025 a cheap way to fuse specialist models into one multitask model. Merging works on task vectors, the weight shifts each fine-tune produced, but the vectors clash where they overlap and pile up unevenly. Their framework, CABS, prunes the vectors with two ideas in sequence.

Conflict-aware pruning masks each vector in turn so the survivors claim distinct parameters. Balanced pruning then keeps key weights while spreading what remains evenly across layers. Across tasks and model sizes, the merged models beat prior merging methods in the team's tests, without any retraining.

Catalog record

CREATORZongzhen Yang, Binhang Qi, Hailong Sun, Wenrui Long, Ruobing Zhao, Xiang Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01874

Browse the full library · About the Tower of Babel