LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model

Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen · 2025

Read LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model by Yuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08582.txt.

Description

Yuxuan Hu and co-authors adapt sparse language models efficiently in a 2025 paper. Sparse models switch off most weights, which saves memory at run time. Fine-tuning them with the popular LoRA method is awkward, because naive adapters fill the zeros back in. Masked variants keep the sparsity but pay heavy memory and computing overhead.

Their method, LoRS, trims that cost with two tricks. Weights are recomputed on the fly rather than stored, and the computation graph is rearranged to skip useless work. Adapter starting values are also chosen more cleverly. Together these cuts lower memory and computing needs during fine-tuning. In tests, LoRS beat earlier LoRA variants on sparse models while using fewer resources.

Catalog record

CREATORYuxuan Hu, Jing Zhang, Xiaodong Chen, Zhe Zhao, Cuiping Li, Hong Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-08582

Browse the full library · About the Tower of Babel