Scalable Multi-Task Low-Rank Model Adaptation

Zichen Tian, Antoine Ledent, Qianru Sun · 2026

Read Scalable Multi-Task Low-Rank Model Adaptation by Zichen Tian, Antoine Ledent, Qianru Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01526.txt.

Description

Adapting one large model to many tasks at once often works at small scale, then collapses as tasks pile up. Zichen Tian, Antoine Ledent, and Qianru Sun diagnose that collapse in this 2026 study of low-rank adaptation. They trace it to clashes between task updates and to blurred traits that routing needs.

The team finds that shared knowledge sits in strong components, while uniform rules force those parts apart. Working at block level, rather than on small pieces, cuts gradient clashes a lot with fewer settings. Their fix, mtLoRA, guards strong parts, adapts by blocks, and routes with fine per-dimension weights. Across vision and language suites with many tasks, it beat top rivals while using fewer settings and less training time.

Catalog record

CREATORZichen Tian, Antoine Ledent, Qianru Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62167
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01526

Browse the full library · About the Tower of Babel