LoRE-Merging : Exploring Low-Rank Estimation For Large Language Model Merging

Zehua Liu, Han Wu, Yuxuan Yao, Ruifeng She, Xiongwei Han, Tao Zhong, Mingxuan Yuan · 2025

Read LoRE-Merging : Exploring Low-Rank Estimation For Large Language Model Merging by Zehua Liu, Han Wu, Yuxuan Yao, Ruifeng She, Xiongwei Han, Tao Zhong, Mingxuan Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-10749.txt.

Description

Zehua Liu, Han Wu, and Yuxuan Yao lead a team merging fine-tuned language models by estimating their task vectors at low rank, no base model required. Their paper appeared in two thousand twenty-five. Merging blends specialist models into one without further training. Interference between specialists is the usual failure mode.

The team noticed task vectors concentrate their energy in a few dominant directions. A low-rank estimate therefore keeps the signal and drops much of the clash. They cast merging as an optimization problem built on that estimate. Wide experiments showed less interference and better preservation of each skill. The method advances merging benchmarks without touching training data. Not needing the base model widens where it can be applied. The authors frame low rank as the natural geometry of fine-tuning. That idea may guide merging theory generally.

Catalog record

CREATORZehua Liu, Han Wu, Yuxuan Yao, Ruifeng She, Xiongwei Han, Tao Zhong, Mingxuan Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-10749

Browse the full library · About the Tower of Babel