Model Merging via Data-Free Covariance Estimation

Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau · 2026

Read Model Merging via Data-Free Covariance Estimation by Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01329.txt.

Description

Abdel Hameed and colleagues rework model merging, the cheap trick of blending separately trained models so one model inherits each one's skills. Framing merging as layer-by-layer minimization of interference normally demands covariance matrices estimated from data, which may be unavailable. They show that under certain conditions those matrices can be read off the models' difference matrices instead. That removes the data need and cuts cost. Across vision and language tests, on models from 86 million to 7 billion parameters, their data-free method beats prior data-free methods.

Catalog record

CREATORMarawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59550
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.01329

Browse the full library · About the Tower of Babel