Model Merging via Data-Free Covariance Estimation
Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau · 2026
Read Model Merging via Data-Free Covariance Estimation by Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01329.txt.
Description
Abdel Hameed and colleagues rework model merging, the cheap trick of blending separately trained models so one model inherits each one's skills. Framing merging as layer-by-layer minimization of interference normally demands covariance matrices estimated from data, which may be unavailable. They show that under certain conditions those matrices can be read off the models' difference matrices instead. That removes the data need and cuts cost. Across vision and language tests, on models from 86 million to 7 billion parameters, their data-free method beats prior data-free methods.
Catalog record
| CREATOR | Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo, Colin Raffel, Guillaume Rabusseau |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59550 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.01329