Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging

Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng Tao · 2025

Read Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging by Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng Tao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09522.txt.

Description

Anke Tang and co-authors merge fine-tuned models one at a time as they arrive, with no retraining, in a 2025 machine learning paper. Standard merging wants every model on the table at once, eats memory, and lets tasks interfere. Their method projects each new model's changes onto directions that leave earlier ones untouched, then rescales to keep distances stable.

Memory use stays flat no matter how many models join. On vision-language models, the sequential merger beat simultaneous rivals by five to eight points of average accuracy and held up under different arrival orders. The approach suits settings where new specialist models appear over time.

Catalog record

CREATORAnke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, Dacheng Tao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09522

Browse the full library · About the Tower of Babel