Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng Tao · 2025
Read Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging by Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09522.txt.
Description
Anke Tang and co-authors merge fine-tuned models one at a time as they arrive, with no retraining, in a 2025 machine learning paper. Standard merging wants every model on the table at once, eats memory, and lets tasks interfere. Their method projects each new model's changes onto directions that leave earlier ones untouched, then rescales to keep distances stable.
Memory use stays flat no matter how many models join. On vision-language models, the sequential merger beat simultaneous rivals by five to eight points of average accuracy and held up under different arrival orders. The approach suits settings where new specialist models appear over time.
Catalog record
| CREATOR | Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng Tao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-09522