Activation-Informed Merging of Large Language Models
Amin Heyrani Nobari, Kaveh Alim *, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan · 2025
Read Activation-Informed Merging of Large Language Models by Amin Heyrani Nobari, Kaveh Alim *, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02421.txt.
Description
Amin Heyrani Nobari and five co-authors introduced Activation-Informed Merging in twenty twenty-five, a method for combining fine-tuned language models more wisely. Merging averages the weights of models tuned for different skills, gaining versatility for free. But uniform merging can overwrite weights the base model truly needs.
Their technique consults the models' activation patterns on a task-agnostic calibration set to judge which weights matter, then protects the essential ones during merging, borrowing ideas from continual learning and compression. It layers onto any existing merging method. Across benchmarks, merged models with this protection gained up to forty percent in performance. The authors argue activation space, not just weight space, should guide how models are combined.
Catalog record
| CREATOR | Amin Heyrani Nobari, Kaveh Alim *, Ali ArjomandBigdeli, Akash Srivastava, Faez Ahmed, Navid Azizan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-02421