SLaB: Sparse-Lowrank-Binary Decomposition for Efficient Large Language Models Ziwei Li, Yuang Ma, Yi Kang 1 Abstract The rapid growth of large language models (LLMs) presents significant deployment challenges due to their massive computational and memory demands. While model compression, such as network pruning, offers potential solutions, most existing methods often fail to maintain good performance at high compression ratios. To address this, we propose SLaB, a novel framework that decomposes each linear layer weight into three complementary components: a sparse matrix, a low-rank matrix, and a binary matrix. SLaB eliminates the need for retraining and leverages activation-aware pruning scores to guide the decomposition process. Experiments on Llama-family models demonstrate that SLaB achieves state-of-the-art performance, reducing perplexity by up to 36 % 36\% compared to existing methods at 50 % 50\% compression and improving accuracy by up to 8.98 % 8.98\% over the baseline on zero-shot tasks. I Introduction In recent years, large language models (LLMs) like GPT [ 19 , 3 ] and BERT [ 7 ] have driven significant progress in artificial intelligence. However, their large size and high computational demands make them difficult to deploy on resource-constrained devices. Model compression [ 13 , 15 , 10 , 9 ] , such as network pruning [ 16 , 14 ] , is an important solution that can effectively remove redundant parameters. However, pruning LLMs during training is computationally expensive. Many studies [ 11 , 22 , 1 , 30 ] investigate one-shot pruning approaches for LLMs, while one-shot pruning causes significant accuracy degradation at high sparsity levels. Meanwhile, low-rank decomposition like ASVD [ 28 ] and SVD-LLM [ 26 ] offers a hardware-friendly approach. Combining sparsity with low-rank approximation may achieve higher accuracy at a high compression ratio. But we experimented with simply combining a sparsification method with a low-rank matrix, which yields poor results, as shown in Figure 1 . Fig. 1: Compression of the Llama-2 7B model [ 24 ] using only low-rank and sparse matrices: perplexity comparison on the WikiText-2 dataset under different rank settings at a 50\% compression ratio. To solve these problems, we propose Sparse-Lowrank-Binary Decomposition (SLaB) , a novel pruning framework that combines low-rank and sparsity while using binary components as much as possible to simplify computations. More details are shown in Figure 2 . Our key insight is to decompose each linear layer weight \mathbf{W} into three matrices: \mathbf{W}=\mathbf{W}_{\text{S}}+\mathbf{W}_{\text{L}}\odot\mathbf{W}_{\text{B}}, (1) where \odot denotes the Hadamard product, \mathbf{W}_{\text{S}} , \mathbf{W}_{\text{B}} and \mathbf{W}_{\text{L}} denote the sparse matrix, low-rank matrix, and binary matrix, respectively. \mathbf{W}_{\text{B}}\in\{+1,-1\} has only binary elements which is hardware-friendly. The fundamental idea of SLaB is to sparsify weight matrices in every linear layer of LLM while using an approach to compensate for the performance loss from sparsity. In SLaB, the original weight matrix can be replaced with a sparse matrix plus a low-rank matrix, which is used to compensate for loss. And finally, combining a binary matrix with a low-rank matrix can significantly reduce the required rank. Our contributions are as follows: (1) we propose a novel decomposition method to decompose weight matrix in a linear layer into 3 matrices: a sparse matrix, a low-rank matrix, and a binary matrix to achieve weight compression; (2) we present an analysis to derive an algorithm for constructing these three matrices and (3) we evaluate SLaB on the widely adopted Llama-family models. Our results demonstrate that SLaB can retain strong performance without any training and outperform previous state-of-the-art pruning methods [ 11 , 22 ] . Fig. 2: Overview of the SLaB framework. II Methodology This section introduces SLaB. First, an optimization strategy is employed to transform the complex optimization problem into a single-matrix optimization task, with its optimization process being analyzed. Subsequently, the complete workflow is presented. Finally, key parameters are discussed. Let the shape of weight matrix \mathbf{W} be (D_{\text{out}},D_{\text{in}}) , and the shape of its corresponding activation matrix \mathbf{X} be (N\times L,D_{\text{in}}) , where N and L are the batch and sequence dimensions, respectively. II-A Algorithm II-A 1 Pruning Method Figure 2 provides an overview of the SLaB framework. At the weight level, to obtain \mathbf{W}_{\text{S}} , \mathbf{W}_{\text{B}} , and \mathbf{W}_{\text{L}} in ( 1 ), we employ an alternating optimization strategy by fixing the other matrices during each update similar to the approach in [ 27 , 4 , 31 ] . Repeating this process for several iterations is sufficient for convergence. And at the layer level, we employ the layer-wise pruning approach [ 11 , 22 ] , which, in one-shot pruning, involves performing the following steps layer by layer: (1) forward propagation, (2) pruning, and (3) updating the layer’s output after pruning. II-A 2 The Optimization of \mathbf{W}_{\text{S}} Given the integration of activation influences into the weight matrices via the Wanda [ 22 ] technique, the optimization of \mathbf{W}_{\text{S}} becomes straightforward. By fixing the remaining components and assuming \mathbf{W}-\mathbf{W}_{\text{L}}\odot\mathbf{W}_{\text{B}}=\mathbf{Y}_{\text{S}} , pruning is performed based on the magnitude of scoring matrix \mathbf{S} , where \mathbf{S}_{ij}=|\mathbf{Y}_{\text{S},ij}|\cdot\|\mathbf{X}_{j}\|_{2} . II-A 3 The Optimization of \mathbf{W}_{\text{B}} Similarly, by fixing the remaining components and introducing the assumption (\mathbf{W}-\mathbf{W}_{\text{S}})\oslash\mathbf{W}_{\text{L}}=\mathbf{Y}_{\text{B}} , where \oslash denotes element-wise division, the optimization objective can be reformulated as \mathop{\arg\min}_{\mathbf{W}_{\text{B}}}\|\mathbf{Y}_{\text{B}}-\mathbf{W}_{\text{B}}\|_{2}^{2} . Furthermore, the following lemma and proposition can be proven: Lemma 1 . If w_{0}\leq w_{1}\leq\cdots\leq w_{n-1},n\geq 2 , \tilde{a} and \tilde{b} are the minimizers of f(a,b)=\sum_{k=0}^{n-1}\min\left\{(w_{k}-a)^{2},(w_{k}-b)^{2}\right\} . Suppose a\leq b , there exists t\in\{0,1,\cdots,n-2\} such that w_{t}\leq\frac{\tilde{a}+\tilde{b}}{2}\leq w_{t+1},\tilde{a}=\frac{\sum_{k=0}^{t}w_{k}}{t+1},\tilde{b}=\frac{\sum_{k=t+1}^{n-1}w_{k}}{n-t-1}. (2) Proof. Clearly, \tilde{a} and \tilde{b} exist; we proceed by contradiction. Furthermore, the conclusion is trivial if either all w_{k} are identical or \tilde{a}=\tilde{b} . Moreover, the minimum can only be attained when a and b take values within the interval [w_{0},w_{n-1}] . In the following, we assume a,b\in[w_{0},w_{n-1}] . Without loss of generality, suppose \tilde{a}<\sum_{k=0}^{t}w_{k}/(t+1)\leq w_{n-1} which implies that there exists t satisfying w_{t}\leq(\tilde{a}+\tilde{b})/2f\left(\min\left\{2w_{t+1}-\tilde{b},\frac{\sum_{k=0}^{t}w_{k}}{t+1}\right\},\tilde{b}\right). (3) This contradicts the assumption. Similar arguments apply to the remaining cases, thus completing the proof of the lemma. ∎ Proposition 1 . If a random matrix \mathbf{X}\sim F possesses a symmetric probability density function \phi(x)=\phi(-x) , and the matrix \mathbf{W}_{\text{B}}\in\{a,b\} is a binary-valued matrix, then a suboptimal solution to the optimization problem \mathop{\arg\min}_{\mathbf{W}_{\text{B}}}\|\mathbf{X}-\mathbf{W}_{\text{B}}\|_{2}^{2} (4) satisfies a+b=0 . Proof. Based on the conclusion of Lemma 1 , a suboptimal solution to ( 4 ) satisfies: w_{t}\leq\frac{\mathbb{E}[\mathbf{X}|\mathbf{X}\leq w_{t}]+\mathbb{E}[\mathbf{X}|\mathbf{X}\geq w_{t+1}]}{2}\leq w_{t+1}. (5) Assuming w_{t}\approx w_{t+1}=s , we obtain that \mathbb{E}[\mathbf{X}|\mathbf{X}\leq s]+\mathbb{E}[\mathbf{X}|\mathbf{X}\geq s]=2s . Clearly, when s=0 , since \phi(x)=\phi(-x) , ( 5 ) holds. Therefore, s=0 is a suboptimal solution, which implies a+b=0 is a suboptimal solution. ∎ Noting that our optimization process does not favor positive or negative values—together with the common assumption that the original linear layer weights \mathbf{W} follow a zero-mean distribution, we apply Proposition 1 and find that using a binary matrix \mathbf{W}_{\text{B}} with values from the set \{a,-a\} constitutes a suboptimal solution. Given the Hadamard product relationship between \mathbf{W}_{\text{B}} and \mathbf{W}_{\text{L}} , the scaling factors of \mathbf{W}_{\text{B}} can be absorbed into \mathbf{W}_{\text{L}} . Therefore, \mathbf{W}_{\text{B}} is constrained to the binary set \{+1,-1\} . II-A 4 The Optimization of \mathbf{W}_{\text{L}} Similarly, with the assumption that (\mathbf{W}-\mathbf{W}_{\text{S}})\oslash\mathbf{W}_{\text{B}}=\mathbf{Y}_{\text{L}} , the optimization problem reduces to \mathop{\arg\min}_{\mathbf{W}_{\text{L}}}\|\mathbf{Y}_{\text{L}}-\mathbf{W}_{\text{L}}\|_{2}^{2} . Guided by the Eckart–Young theorem [ 8 ] , the optimal solution with a specified rank constraint is obtained by truncated singular value decomposition (SVD). II-A 5 The Joint Optimization of \mathbf{W}_{\text{B}} and \mathbf{W}_{\text{L}} Let \mathbf{W}-\mathbf{W}_{\text{S}}=\mathbf{Y}_{\text{BL}} . To alternately optimize \mathbf{W}_{\text{B}} and \mathbf{W}_{\text{L}} , we naturally adopt the following straightforward initialization: \displaystyle\mathbf{W}_{\text{B}} \displaystyle=\text{sign}(\mathbf{Y}_{\text{BL}}), (6) \displaystyle\mathbf{W}_{\text{L}} \displaystyle=\mathbf{U}\mathbf{V}^{\mathrm{T}}\;\text{with}\;\mathbf{U}=\sqrt{\sigma_{0}}\mathbf{u}_{0},\mathbf{V}=\sqrt{\sigma_{0}}\mathbf{v}_{0}, \displaystyle|\mathbf{Y}_{\text{BL}}| \displaystyle=\sum_{k=0}^{\min\{D_{\text{out}},D_{\text{in}}\}}\sigma_{k}\mathbf{u}_{k}\mathbf{v}_{k}^{\mathrm{T}}, where \text{sign}(\cdot) denotes a function that judges the input as positive or negative, non-negative numbers are denoted as 1 while negative numbers are denoted as 0, \mathbf{U} and \mathbf{V} are computed from \mathbf{Y}_{\text{BL}} using a rank-1 truncated SVD. Proposition 2 . If \mathbf{U}\in\mathbb{R}^{(D_{\text{out}},1)} and \mathbf{V}\in\mathbb{R}^{(D_{\text{in}},1)} are the rank-1 truncated SVD results of |\mathbf{Y}_{\text{BL}}| , satisfying \mathbf{W}_{\text{L}}=\mathbf{U}\mathbf{V}^{\mathrm{T}} . Then, every element of \mathbf{W}_{\text{L}} is non-negative, i.e. \lvert\mathbf{W}_{\text{L}}\rvert=\mathbf{W}_{\text{L}} . Proof. Consider the Frobenius norm of the difference between each of these matrices and |\mathbf{Y}_{\text{BL}}| . Clearly, both \mathbf{W}_{\text{L}} and |\mathbf{W}_{\text{L}}| are rank-1 matrices, so we have (due to the Eckart-Young theorem): \|\mathbf{W}_{\text{L}}-|\mathbf{Y}_{\text{BL}}|\|_{F}^{2}\leq\||\mathbf{W}_{\text{L}}|-|\mathbf{Y}_{\text{BL}}|\|_{F}^{2}, (7) where equality holds if and only if \mathbf{W}_{\text{L}}=|\mathbf{W}_{\text{L}}| . However, by the triangle inequality, \|\mathbf{W}_{\text{L}}-|\mathbf{Y}_{\text{BL}}|\|_{F}^{2}\geq\||\mathbf{W}_{\text{L}}|-|\mathbf{Y}_{\text{BL}}|\|_{F}^{2}. (8) So we have \mathbf{W}_{\text{L}}=|\mathbf{W}_{\text{L}}| and the proposition is proven. ∎ When using the initialization in ( 6 ), if \mathbf{W}_{\text{B}} is fixed as \text{sign}(\mathbf{Y}_{\text{BL}}) , the optimal solution for \mathbf{W}_{\text{L}} is given by \mathbf{U}\mathbf{V}^{\mathrm{T}} . Conversely, if \mathbf{W}_{\text{L}} is fixed, Proposition 2 guarantees its non-negativity, and thus the optimal choice for \mathbf{W}_{\text{B}} is \text{sign}(\mathbf{Y}_{\text{BL}}) . So ( 6 ) yields a suboptimal solution for both \mathbf{W}_{\text{B}} and \mathbf{W}_{\text{L}} . Adopting the initialization from ( 6 ) significantly reduces the computational overhead of alternating optimization while maintaining efficacy. And the rank-1 setting will be discussed in the Section II-B3 . II-B Parameters II-B 1 Sparsity Let the original matrix \mathbf{W} , along with \mathbf{W}_{\text{S}} and \mathbf{W}_{\text{L}} , be b -bit wide data (e.g. b=16 for FP16). Suppose \mathbf{W}_{\text{S}} contains k non-zero elements, \mathbf{W}_{\text{B}} is a binarized matrix where each element is stored using 1 bit, and \mathbf{W}_{\text{L}} is a rank-1 matrix expressible as the product of two vectors with shapes (D_{\text{out}},1) and (1,D_{\text{in}}) , respectively. Thus, the compression ratio ( CR ) of SLaB is: \text{CR}=1-\frac{(\overbrace{bk}^{\mathbf{W}_{\text{S}}}+\overbrace{D_{\text{out}}D_{\text{in}}}^{\mathbf{W}_{\text{B}}}+\overbrace{b(D_{\text{out}}+D_{\text{in}})}^{\mathbf{W}_{\text{L}}})}{bD_{\text{out}}D_{\text{in}}}. (9) Therefore, the percentage of non-zero elements retained in the sparse matrix \mathbf{W}_{\text{S}} relative to the total elements can be derived: \frac{k}{D_{\text{out}}D_{\text{in}}}=1-\text{CR}-\frac{1}{b}-\frac{1}{D_{\text{out}}}-\frac{1}{D_{\text{in}}}. (10) II-B 2 Comparison Group Size Following [ 11 , 22 ] , we use comparison groups of size (1,D_{\text{in}}) (i.e., the number of elements in each group N_{g}=D_{\text{in}} ), within which we prune by comparing the score matrices. This results in \lfloor kN_{g}/(D_{\text{out}}D_{\text{in}})\rfloor=\lfloor k/D_{\text{out}}\rfloor non-zero elements retained per group. For the semi-structured pruning method [ 18 ] like 2:4 and 4:8, we first apply semi-structured pruning and then perform group-wise pruning to reach the target sparsity. II-B 3 Rank and Iterations As shown in Figure 3 , when all layers are pruned at a 50\% compression ratio, increasing the rank from 0 (corresponding to the Wanda method) to 1 significantly reduces the average Frobenius norm difference between the compressed and original weight. However, as the rank continues to increase, the reduction becomes much more gradual. Therefore, we adopt the rank-1 setting for simplicity. Fig. 3: Variation of the average Frobenius norm difference between compressed and original layers with respect to rank. Experiments are conducted on the Llama-2 7B model [ 24 ] with a 50\% compression ratio. Algorithm 1 The SLaB algorithm. 0: Weight \mathbf{W}\in\mathbb{R}^{(D_{\text{out}},D_{\text{in}})} , activation obtained through the forward propagation of the preceding layer \mathbf{X}\in\mathbb{R}^{(D_{\text{out}},D_{\text{in}})} , number of iterations s , compression ratio CR , the bit-width of the output matrix (excluding the binary matrix) b 0: Sparse weight \mathbf{W}_{\text{S}}\in\mathbb{R}^{(D_{\text{out}},D_{\text{in}})} , matrix \mathbf{U}\in\mathbb{R}^{(D_{\text{out}},1)},\mathbf{V}\in\mathbb{R}^{(D_{\text{in}},1)} , binary matrix \mathbf{W}_{\text{B}}\in\{1,-1\}^{(D_{\text{out}},D_{\text{in}})} 1: Init \mathbf{W}_{\text{S}}\leftarrow\mathbf{0} , \mathbf{U}\leftarrow\mathbf{0} , \mathbf{V}\leftarrow\mathbf{0} , \mathbf{W}_{\text{B}}\leftarrow\mathbf{0} 2: \text{sparsity}\leftarrow 1-\text{CR}-\frac{1}{b}-\frac{1}{D_{\text{out}}}-\frac{1}{\text{in}} 3: \mathbf{S}_{\text{X}}\leftarrow\text{diag}\left(\sqrt{\mathbf{X}^{\mathrm{T}}\mathbf{X}}\right)\in\mathbb{R}^{(1,D_{\text{in}})} 4: for step t=1 to s do 5: \mathbf{W}_{\text{B}}\leftarrow\text{sign}(\mathbf{W}-\mathbf{W}_{\text{S}}) 6: \mathbf{U},\mathbf{V}\leftarrow\sqrt{\sigma_{0}}\mathbf{u}_{0},\sqrt{\sigma_{0}}\mathbf{v}_{0} \text{with}\;|\mathbf{W}-\mathbf{W}_{\text{S}}|=\sum_{k=0}^{\min\{D_{\text{out}},D_{\text{in}}\}}\sigma_{k}\mathbf{u}_{k}\mathbf{v}_{k}^{\mathrm{T}} 7: \mathbf{S}\leftarrow\left|\mathbf{W}-\mathbf{U}\mathbf{V}^{\mathrm{T}}\odot\mathbf{W}_{\text{B}}\right|\odot\mathbf{S}_{\text{X}} 8: \mathbf{W}_{\text{S}}\leftarrow\text{HardThreshold}(\mathbf{S},\text{sparsity})\oslash\mathbf{S}_{\text{X}} 9: end for 10: return \mathbf{W}_{\text{S}},\mathbf{U},\mathbf{V},\mathbf{W}_{\text{B}} In addition to the parameters mentioned above, the number of iterations s in the alternating optimization algorithm also affects the outcome. We will conduct hyperparameter exploration on this in the experimental section. II-C Full Algorithm Pseudocode Algorithm 1 presents the complete pruning process of a linear layer using SLaB. The HardThreshold function prunes a given matrix by magnitude according to a specified CR. Comparison group size is not explicitly shown in Algorithm 1 . III Experiments III-A Experiments Setup III-A 1 Models and Baselines We perform pruning and analysis on the Llama-2 7B [ 24 ] , Llama-3 8B, and Llama-3.2 1B [ 12 ] models using PyTorch and the Transformers library provided by Hugging Face. The SLaB is a one-shot pruning method, so that we compare results against two SOTA, also one-shot approaches: SparseGPT [ 11 ] and Wanda [ 22 ] . III-A 2 Calibration Data Consistent with the approach in SparseGPT, we construct the calibration dataset using 128 sequences of length 2048, which are randomly sampled from the first shard of the C4 training set [ 20 ] . To ensure a fair comparison, all pruning algorithms employ this identical calibration dataset (where applicable). III-A 3 Evaluation We evaluate the performance of the pruned models across both zero-shot tasks and language modeling capabilities. We select seven tasks include ARC-C, ARC-E [ 6 ] , BoolQ [ 5 ] , HellaSwag [ 29 ] , PIQA [ 2 ] , RTE [ 25 ] , and WinoGrande [ 21 ] from the LM-Eval-Harness [ 23 ] for zero-shot evaluation. Language modeling capability is quantified by measuring perplexity on the WikiText-2 dataset [ 17 ] . III-A 4 Parameters Like SparseGPT and Wanda, we focus our pruning efforts on all linear layers, excluding the first embedding layers and the final classification head. The pruning is carried out in different categories: unstructured/structured sparsity and CR (Compression Ratio). For unstructured sparsity, we evaluate performance with \text{CR}=50\%,60\%,70\%,80\% ; For structured sparsity, we evaluate two specific sparsity patterns: 2:4 and 4:8, both configured with \text{CR}=50\% . Regarding the comparison group size, we set it to (1,D_{\text{in}}) . The number of iterations for SLaB is set to 20. TABLE I: Comparison of perplexity on WikiText-2 and average zero-shot accuracies ( \% ) at different compression rates. Method Sparsity( CR ) Llama-3.2 1B Llama-2 7B Llama-3 8B ppl 2 \downarrow acc 2 \uparrow ppl \downarrow acc \uparrow ppl \downarrow acc \uparrow Dense 0\% 9.06 58.3 5.12 68.4 5.75 72.9 SparseGPT US 1 ( 50\% ) 18.09 51.2 6.52 63.9 8.73 66.9 Wanda US ( 50\% ) 21.37 48.5 6.45 64.0 9.15 65.4 SLaB US ( 50\% ) \boldsymbol{11.57} \boldsymbol{55.8} \boldsymbol{5.49} \boldsymbol{66.2} \boldsymbol{6.67} \boldsymbol{70.7} SparseGPT 4:8 ( 50\% ) 21.64 48.0 7.94 61.6 11.33 61.3 Wanda 4:8 ( 50\% ) 40.47 44.3 8.03 60.8 13.41 65.4 SLaB 4:8 ( 50\% ) \boldsymbol{12.43} \boldsymbol{52.5} \boldsymbol{5.61} \boldsymbol{65.3} \boldsymbol{6.93} \boldsymbol{70.1} SparseGPT 2:4 ( 50\% ) 30.14 46.5 10.37 58.9 14.81 57.4 Wanda 2:4 ( 50\% ) 79.06 41.3 11.40 55.6 22.52 52.2 SLaB 2:4 ( 50\% ) \boldsymbol{14.02} \boldsymbol{52.5} \boldsymbol{5.77} \boldsymbol{65.3} \boldsymbol{7.32} \boldsymbol{70.1} SparseGPT US ( 60\% ) 44.43 45.2 9.52 58.7 14.45 59.0 Wanda US ( 60\% ) 71.92 41.5 10.01 57.1 21.37 53.3 SLaB US ( 60\% ) \boldsymbol{14.73} \boldsymbol{51.7} \boldsymbol{5.85} \boldsymbol{65.0} \boldsymbol{7.54} \boldsymbol{69.2} SparseGPT US ( 70\% ) 132.62 40.8 24.84 47.6 38.52 46.9 Wanda US ( 70\% ) 412.56 37.4 71.58 39.9 107.00 39.3 SLaB US ( 70\% ) \boldsymbol{26.06} \boldsymbol{46.6} \boldsymbol{6.88} \boldsymbol{62.2} \boldsymbol{10.35} \boldsymbol{64.3} SparseGPT US ( 80\% ) 471.50 38.5 111.27 37.6 183.01 39.5 Wanda US ( 80\% ) 1.34\mathrm{e}{4} 36.6 3.60\mathrm{e}{3} 36.1 1.25\mathrm{e}{3} 37.2 SLaB US ( 80\% ) \boldsymbol{113.77} \boldsymbol{41.5} \boldsymbol{13.63} \boldsymbol{53.4} \boldsymbol{27.02} \boldsymbol{49.3} 1 US refers to unstructured sparsity. 2 ppl refers to perplexity (lower is better) and acc refers to accuracy (higher is better). III-B Results Table I presents perplexity on the WikiText-2 dataset and average accuracies on zero-shot tasks for SLaB and baselines. Using 50\% compression ratio for the Llama-3.2 1B model, SLaB reduces perplexity by 36.04\% and improves accuracy by 8.98\% compared to the best-performing baseline. III-C Hyperparameter Exploration Table II presents the impact of the comparison group size and the number of iterations on the results. TABLE II: Hyperparameter exploration with Llama-2 7B ( \text{CR}=50\% ) model. Comparison \left(1,\displaystyle\frac{D_{\text{in}}}{32}\right) \left(1,\displaystyle\frac{D_{\text{in}}}{16}\right) (1,D_{\text{in}}) (16,D_{\text{in}}) (32,D_{\text{in}}) Group ppl \downarrow 5.516 5.491 5.493 5.544 5.546 acc \uparrow ( \% ) 65.6 65.8 66.2 65.9 66.2 Iterations 1 10 20 30 40 ppl( \downarrow ) 5.678 5.531 5.493 5.480 5.477 III-D Ablation Study As shown in Table III , the incorporation of both the \mathbf{W}_{\text{L}} and \mathbf{W}_{\text{B}} components has a compensatory effect on \mathbf{W}_{\text{S}} , thereby enhancing the model’s accuracy. TABLE III: Ablation study with Llama-2 7B (2:4, \text{CR}=50\% ) model. Accuracy ( \% ) ARC-C ARC-E RTE WinoGrande Avg \mathbf{W}_{\text{S}} 32.0 58.2 53.1 62.0 49.8 \mathbf{W}_{\text{S}}+\mathbf{W}_{\text{L}}(r=16) 33.4 59.2 53.8 64.8 51.2 \mathbf{W}_{\text{S}}+\text{factor}{\textsuperscript{*}}\odot\mathbf{W}_{\text{B}} 42.1 71.8 55.6 68.2 58.2 \mathbf{W}_{\text{S}}+\mathbf{W}_{\text{L}}\odot\mathbf{W}_{\text{B}} 43.2 71.3 57.0 68.6 58.9 * factor refers to the quantization factor vector. IV Conclusion In this work, we propose a novel method called SLaB for compressing LLMs. SLaB replaces weights with a combination of sparse, binary, and low-rank components, achieving better performance without any training. This work provides a new perspective for future research on combining sparsity with other compression techniques in a complementary manner.