LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization

Xujia Wang Yunjia Qi Bin Xu · 2025

Read LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization by Xujia Wang Yunjia Qi Bin Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04487.txt.

Description

Xujia Wang, Yunjia Qi, and Bin Xu present a study from twenty twenty-five on fine-tuning large models by training only a key sub-network. Cheap tuning methods add small low rank blocks to a big model. Those blocks cost many sums and can cap the gains. Their method, LoSiA, hunts for the weights that matter most, then tunes just those.

The hunt uses how sparse the gradient is across the network. Tuning the picked sub-network gives high rank change with no extra matrix sums. A pro form trims training time by about a fourth next to a known rival. Tests on expert tasks and plain reason tasks show small drops from full tuning. The method also cuts how much old skill is lost in long runs.

Catalog record

CREATORXujia Wang Yunjia Qi Bin Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04487

Browse the full library · About the Tower of Babel