SALR: Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models
Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu · 2026
Read SALR: Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models by Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16991.txt.
Introduction
Longteng Zhang and a team of nine co-authors trim big chatbots for new tasks at low cost. Fine-tune work oft trains vast weight counts. That cost blocks use where gear is scarce. Low-rank tricks cut the train count by a lot. Yet base weights still hog space and time. Their 2026 preprint adds sparse sense to those tricks.
Their build, SALR, pairs low rank with sparse picks. Picks keep key weights and skip the rest. Apps on phones and small clouds gain most. The team aims to hold skill while costs fall. Cheap tune paths widen who can shape big bots.
Catalog record
| CREATOR | Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16991
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SALR: Sparsity-Aware Low-Rank Representation for Efficient Fine-Tuning of Large Language Models Longteng Zhang, Sen Wu, Shuai Hou, Zhengyu Qing, Zhuo Zheng, Danning Ke, Qihong Lin, Qiang Wang, Shaohuai Shi, Xiaowen Chu Abstract Adapting large pre-trained language models to downstream tasks often entails fine-tuning millions of parameters or deploying costly dense weight updates, which hinders their use in resource-constrained environments. Low-rank Adaptation (LoRA) reduces trainable parameters by factorizing weight updates, yet the underlying dense weights still impose high storage and computation costs. Magnitude-based pruning can yield sparse models but typically degrades LoRA’s performance when applied naively. In this paper, we introduce SALR (Sparsity-Aware Low-Rank Representation), a novel fine-tuning paradigm that unifies low-rank adaptation with sparse pruning under a rigorous mean-squared-error framework. We prove that statically pruning only the frozen base weights minimizes the pruning error bound, and we recover the discarded residual information via a truncated-SVD low-rank adapter, which provably reduces per-entry MSE by a factor of ( 1 − r / min ( d , k ) ) (1-r/\min(d,k)) . To maximize hardware efficiency, we fuse multiple low-rank adapters into a single concatenated GEMM, and we adopt a bitmap-based encoding with a two-stage pipelined decoding+GEMM design to achieve true model compression and speedup. Empirically, SALR attains 50% sparsity on various LLMs while matching the performance of LoRA on GSM8K and MMLU, reduces model size by 2 × 2\times , and delivers up to a 1.7 × 1.7\times inference speedup. Introduction The rapid growth of large-scale neural networks has driven remarkable advances across natural language processing, computer vision, and other machine learning domains ( 24 ; 6 ; 31 ; 4 ; 25 ) . However, adapting these massive models to downstream tasks often requires fine-tuning millions or even billions of parameters, posing significant challenges for deployment in resource-constrained environments ( 22 ; 18 ; 20 ) . Low-Rank Adaptation (LoRA) ( 12 ) has emerged as a lightweight fine-tuning paradigm, representing weight updates as a product of two much smaller matrices. While LoRA drastically reduces the number of trainable parameters, the underlying dense structure of the original weights still incurs substantial memory and computation overhead ( 5 ; 33 ) . Model pruning offers a complementary approach by eliminating redundant weights to yield sparse networks with reduced storage and faster inference ( 11 ; 9 ; 26 ; 10 ) . Ideally, magnitude-based pruning applied to a LoRA-fine-tuned model should combine the benefits of sparsity and low-rank adaptation. However, pruning can disrupt the carefully learned low-rank subspace, and existing dynamic masking strategies often lead to significant performance degradation ( 14 ) . To guide effective pruning in the LoRA setting, we develop a unified theoretical framework that quantifies the mean-squared error (MSE) induced by different pruning schemes. Our analysis reveals that a static mask applied solely to the frozen base weights achieves the lowest error bound among both static and dynamic approaches, laying the foundation for our Sparsity-Aware Low-Rank Representation fine-tuning (SALR) method. Figure 1: Memory-accuracy trade-off on the GSM8K ( 3 ) benchmark for Llama3-8B ( 6 ) , fine-tuned on MetaMath ( 32 ) . At 50% sparsity, SALR maintains the dense LoRA ( 12 ) baseline accuracy (79.5%) while reducing model size from 15.5 GB to 7.98 GB. In contrast, LoSA ( 14 ) at 50% sparsity suffers a drop to 71.4% accuracy. Although static pruning of the base weights minimizes theoretical error, it discards valuable information residing in the pruned elements. To address this loss, SALR introduces a sparsity preservation pruning strategy: rather than permanently zeroing out pruned entries, we capture their residual in an auxiliary low-rank adapter. By applying a truncated singular value decomposition (SVD) to the sparse residual, we have a compact rank- r correction that retains essential information without inflating the parameter count. We further show that this low-rank correction provably reduces the per-entry MSE by a factor of \bigl(1-\tfrac{r}{\min(d,k)}\bigr) , where d and k are the input and output dimensions. However, in this setting, SALR requires the deployment of two or more low-rank adapters, one corresponding to the LoRA fine-tuning module and others serving as sparsity preservation modules. Performing the small matrix multiplications for each adapter sequentially results in under-utilization of modern hardware accelerators. To address this inefficiency, SALR adopts an adapter concatenation scheme: by stacking the low-rank matrices along their rank dimension, all adapter updates are consolidated into a single, larger GEMM (general matrix-matrix multiplication) operation on a shared input. This fusion reduces kernel-launch overhead and maximizes computational throughput when multiple adapters operate on the same input vector. In recent LoRA-based pruning studies, although most works claim to achieve a specified sparsity ratio and the associated speedup, almost none address the challenge of actual model size compression. Thus, the achieved sparsity does not necessarily translate into reduced model size ( 14 ; 16 ; 1 ; 17 ) . This limitation is a major reason why sparse methods lag behind quantization in model compression ( 8 ; 30 ) . Therefore, to enable effective model compression via pruning, SALR integrates a bitmap-based encoding for the pruned base weights along with a two-stage pipelined design. By employing bitmap encoding, SALR efficiently stores the dense elements of sparse weights and enables high-performance bitmap decoding. The two-stage pipeline further alleviates decoding overhead: in the first stage, byte-level masks and lookup tables reconstruct sparse submatrices efficiently, while the LoRA module participates in GEMM computation; concurrently, the second stage feeds these reconstructed blocks into high-performance GEMM kernels. In this manner, the two-stage pipeline sustains compute-bound density throughout all computation phases. In summary, our contributions are: • We develop a unified, MSE-based theoretical framework for pruning in LoRA-fine-tuned models and prove that applying a static mask to the frozen base weights yields the lowest error bound. • We introduce SALR, a sparsity-preservation pruning method that captures pruned-weight residuals via a truncated-SVD low-rank adapter and provably reduces per-entry MSE. • We propose an adapter concatenation scheme that stacks all low-rank adapters into a single GEMM operation, minimizing kernel-launch overhead and maximizing hardware accelerator utilization. • We design a practical compression and deployment pipeline using bitmap encoding for actual model-size reduction and a two-stage, pipelined decoding+GEMM approach to sustain compute-bound throughput. Preliminary In this section, we present the essential background and theoretical underpinnings necessary to understand the challenges and methodologies associated with Sparsity-Aware Low-Rank Representation fine-tuning (SALR). Let the weight matrix be denoted as W\in\mathbb{R}^{d\times k} , where d represents the number of input features and k denotes the number of output classes. The magnitude-based pruning operation can be formulated as \hat{W}_{ij}=\begin{cases}W_{ij},&|W_{ij}|>T_{p},\\ 0,&|W_{ij}|\leq T_{p},\end{cases} where T_{p} is the pruning threshold determined by the desired sparsity ratio p . In this manner, pruning can significantly reduce the memory footprint and computational cost of the model, thereby enabling the deployment of large-scale models in resource-constrained environments. In the following, we analyze the error introduced by pruning and discuss how it can be mitigated through low-rank adaptation techniques. Theorem 1 . Let W\sim\mathcal{N}(0,\sigma^{2}) and for a given pruning ratio p\in[0,1) we choose the threshold T_{p} such that P\bigl(|W|\leq T_{p}\bigr)=p,\quad\Longrightarrow\quad T_{p}=\sigma\Phi^{-1}\Bigl(\tfrac{1+p}{2}\Bigr). Then the mean-squared error (MSE) of this pruning is \mathrm{MSE}(p)=\mathbb{E}\bigl[(W-\hat{W})^{2}\bigr]=2\sigma^{2}\Bigl[\Phi(t_{p})-\tfrac{1}{2}-t_{p}\varphi(t_{p})\Bigr], where t_{p}=\Phi^{-1}\bigl(\tfrac{1+p}{2}\bigr) and \varphi(t)=\tfrac{1}{\sqrt{2\pi}}e^{-t^{2}/2} is the standard normal PDF. Based on