LSR-Adapt: Ultra-Efficient Parameter Tuning with Matrix Low Separation Rank Kernel Adaptation

Xin Li, Anand D. Sarwate · 2025

Read LSR-Adapt: Ultra-Efficient Parameter Tuning with Matrix Low Separation Rank Kernel Adaptation by Xin Li, Anand D. Sarwate (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13568.txt.

Description

Tuning a huge trained model for a new task can be done by training a small patch instead of the whole net. Xin Li and Anand Sarwate shrink that patch far more in 2025. Their trick borrows an old idea from number work: some matrices can be split into a few simple layers, a form with low separation rank.

Common patches already use a low-rank split to cut their size. As models grow, even that patch grows large. The new kernel wraps the patch's matrices in the layered form, which cuts the count of trainable values to about half of the past best. Tests report state-of-the-art scores at that smaller size. The gain is plain thrift: less memory, less time, and a patch that is cheap to store per task.

Catalog record

CREATORXin Li, Anand D. Sarwate
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13568

Browse the full library · About the Tower of Babel