Parameter-Efficient Interventions for Enhanced Model Merging

Marcin Osial, Daniel Marczak, Bartosz Zieliński · 2024

Read Parameter-Efficient Interventions for Enhanced Model Merging by Marcin Osial, Daniel Marczak, Bartosz Zieliński (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17023.txt.

Description

Marcin Osial and co-authors improve the craft of merging several fine-tuned models into one that does all their jobs. Model merging avoids retraining on every task's data, but merged models often blur each task's features — a representation bias that drags performance down. Their 2024 paper proposes IntervMerge, which corrects the blur with small task-specific interventions inside the shared model.

A lighter variant, mini-interventions, adjusts only part of the representation, trimming the extra parameters further. Across experiments, the method beat state-of-the-art merging approaches while adding fewer parameters than they do. The appeal is frugality: one model to ship, many skills inside. Machine learning practitioners get a cheaper path from a zoo of specialists to a single generalist.

Catalog record

CREATORMarcin Osial, Daniel Marczak, Bartosz Zieliński
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17023

Browse the full library · About the Tower of Babel