LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model
Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen · 2025
Read LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model by Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08582.txt.
Description
Yuxuan Hu and co-authors adapt sparse language models efficiently in a 2025 paper. Sparse models switch off most weights, which saves memory at run time. Fine-tuning them with the popular LoRA method is awkward, because naive adapters fill the zeros back in. Masked variants keep the sparsity but pay heavy memory and computing overhead.
Their method, LoRS, trims that cost with two tricks. Weights are recomputed on the fly rather than stored, and the computation graph is rearranged to skip useless work. Adapter starting values are also chosen more cleverly. Together these cuts lower memory and computing needs during fine-tuning. In tests, LoRS beat earlier LoRA variants on sparse models while using fewer resources.
Catalog record
| CREATOR | Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-08582