LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization
Xujia Wang Yunjia Qi Bin Xu · 2025
Read LoSiA: Efficient High-Rank Fine-Tuning via Subnet Localization and Optimization by Xujia Wang Yunjia Qi Bin Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04487.txt.
Description
Xujia Wang, Yunjia Qi, and Bin Xu present a study from twenty twenty-five on fine-tuning large models by training only a key sub-network. Cheap tuning methods add small low rank blocks to a big model. Those blocks cost many sums and can cap the gains. Their method, LoSiA, hunts for the weights that matter most, then tunes just those.
The hunt uses how sparse the gradient is across the network. Tuning the picked sub-network gives high rank change with no extra matrix sums. A pro form trims training time by about a fourth next to a known rival. Tests on expert tasks and plain reason tasks show small drops from full tuning. The method also cuts how much old skill is lost in long runs.
Catalog record
| CREATOR | Xujia Wang Yunjia Qi Bin Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-04487