Activation-Informed Merging of Large Language Models

Amin Heyrani Nobari, Kaveh Alim *, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan · 2025

Read Activation-Informed Merging of Large Language Models by Amin Heyrani Nobari, Kaveh Alim *, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02421.txt.

Description

Amin Heyrani Nobari and five co-authors introduced Activation-Informed Merging in twenty twenty-five, a method for combining fine-tuned language models more wisely. Merging averages the weights of models tuned for different skills, gaining versatility for free. But uniform merging can overwrite weights the base model truly needs.

Their technique consults the models' activation patterns on a task-agnostic calibration set to judge which weights matter, then protects the essential ones during merging, borrowing ideas from continual learning and compression. It layers onto any existing merging method. Across benchmarks, merged models with this protection gained up to forty percent in performance. The authors argue activation space, not just weight space, should guide how models are combined.

Catalog record

CREATORAmin Heyrani Nobari, Kaveh Alim *, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02421

Browse the full library · About the Tower of Babel