R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference

Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, Steven Li · 2025

Read R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference by Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, Steven Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19449.txt.

Introduction

Zhenyu Zhang and co-authors present R-Sparse, a training free way to cut activation cost in large language models in a 2025 paper. Big models are costly at run time, above all on edge devices. Using only a sparse subset of activations can cut compute and memory traffic, but many methods fail on modern activation types or need heavy retraining.

Two observations drive the design. Within one linear layer, the non sparse part of the input acts like a few bias terms, and the full computation can be approximated by mixing input channels with weight singular values. Replacing the linear layers on that basis reaches high sparsity in advanced models without retraining.

Catalog record

CREATORZhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, Steven Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-19449

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference Zhenyu Zhang, Zechun Liu, Yuandong Tian, Harshit Khaitan, Zhangyang Wang, Steven Li Abstract Large Language Models (LLMs), while demonstrating remarkable capabilities across various applications, present significant challenges during inference due to their substantial model size, especially when deployed on edge devices. Activation sparsity offers a promising solution to reduce computation and memory movement, enabling more efficient inference, particularly for small-batch on-device applications. However, current approaches face limitations with non-ReLU activation function, which are foundational to most advanced LLMs, or require heavy continual training. Additionally, the difficulty in predicting active channels and limited achievable sparsity ratios constrain the effectiveness of activation sparsity-based methods. In this paper, we introduce R-Sparse , a training-free activation sparsity approach capable of achieving high sparsity levels in advanced LLMs. We conducted two preliminary investigations into how different components contribute to the output within a single linear layer and found two key observations: (i) the non-sparse components of the input function can be regarded as a few bias terms, and (ii) The full computation can be effectively approximated by an appropriate combination of input channels and weight singular values. Building on this, we replace the linear layers in LLMs with a rank-aware sparse inference method that leverages the sparsity of input channels and singular value components, eliminating the need for active channel prediction like the output sparsity based approaches. Experiments on Llama-2/3 and Mistral models across ten diverse tasks demonstrate that R-Sparse achieves comparable performance at 50% model-level sparsity, resulting in a significant 43% end-to-end efficient improvements with customized kernels. The code is available at https://github.com/VITA-Group/R-Sparse . 1 Introduction Large Language Models (LLMs) have become ubiquitous due to their remarkable capabilities, powering applications from virtual assistants to automated content creation. However, their impressive performance comes with significant computational and memory costs due to their enormous parameter counts. This poses significant challenges for latency-sensitive applications, particularly for deployments on edge devices. To address this, network pruning or sparsity ( Frantar & Alistarh 2023 ; Sun et al. 2023 ; Yin et al. 2023 ; Ma et al. 2023 ) is an effective solution. These strategies operate in a data-independent manner with different levels of pruning granularity, e.g. , unstructured, semi-structured, or structured. While more structured pruning approaches leads to more limited sparsity levels, unstructured sparsity introduces greater challenges for efficient hardware implementation. Recently, activation sparsity ( Liu et al. 2023 ; Mirzadeh et al. 2023 ; Dong et al. 2024 ; Lee et al. 2024 ) has emerged as a promising solution that dynamically loads only the active channels and their corresponding weight rows or columns from off-chip HBM ( NVIDIA 2020 ) to on-chip SRAM, significantly alleviate the latency and memory cost when equipped with optimized system implementations ( Song et al. 2023 ) . Designing activation sparsity functions in a structured, data-dependent way, can make the specified network more hardware-friendly while also achieving higher sparsity levels compared to traditional pruning techniques. Despite the promising progress, several challenges remain: (i) Feasibility for non-ReLU based LLMs : ReLU eliminates the negative part of activations, enabling a lossless approximation when skipping the computation of corresponding channels ( Liu et al. 2023 ) . However, most advanced LLMs now use non-ReLU activations like SiLU ( Elfwing et al. 2018 ) and GELU ( Hendrycks & Gimpel 2016 ) , which retain small negative values, requiring extensive continual pre-training to obtain meaningful activation sparsity ( Song et al. 2024a ; Zhang et al. 2024a ; Mirzadeh et al. 2023 ; Song et al. 2024b ) . Such training process can involve up to 150B tokens, taking approximately one month on 64 A100 GPUs. (ii) Difficulty in Predicting Active Channels : Previous approaches identify critical channels within the hidden activations of MLP blocks, facing significant challenges in predicting the active channels before performing the computation. Common strategies include exploiting the similarity of activated channels across semantically similar tokens ( Dong et al. 2024 ) , leveraging the activations after the gate projection ( Lee et al. 2024 ) , or using a learnable predictor ( Liu et al. 2023 ) , while the accuracy of active channel prediction will highly affect their effectiveness. (iii) Limited Sparsity Levels : For approaches that do not rely on extensive retraining ( Lee et al. 2024 ; Dong et al. 2024 ) , only 50% sparsity within MLP blocks can be achieved, leading to a model-level sparsity of one third. Achieving higher levels of overall sparsity remains a significant challenge. This paper targets a training-free activation sparsity approach that is: (i) feasible for non-ReLU based LLMs; (ii) unaffected by the difficulty of predicting active channels; and (iii) capable of achieving higher sparsity levels. While previous methods focus on output activation sparsity ( Liu et al. 2023 ; Lee et al. 2024 ) , requiring prior prediction of important channels, our approach leverages input activation sparsity, identifying active channels directly from the input without the need of prediction. Furthermore, recent studies ( Mirzadeh et al. 2023 ; Song et al. 2024a ; Song et al. 2024b ) have shown that directly removing the non-sparse components only achieves limited sparsity while with extensive training, sparsity ratios can be pushed to as high as 90%. This sparsity gap raises an natural question: Is the non-sparse portion of the activation truly necessary for maintaining model performance, or can we employ a lightweight strategy to mitigate the non-sparse part without resorting to heavy pre-training? Motivated by this, we apply a multi-phase ReLU function to the non-sparse channels, the corresponding activations will then be rounded to a few discrete values. As the number of discrete values increases from 1 to 2, performance can be significantly improved, even at a sparsity level of 90%. The output components associated with the non-sparse portion can then be approximated by a few bias terms, indicating a low-rank structure for these components. To better understand the low-rank structure, we analyze the importance of each input channel in the activations and each singular value component of the weights, to the output activations. Figure 1: Contributions of each input channel and singular value components. The measurement metric is detailed in Section 3.3 . Results are obtained from Llama-2-7B with 16 training samples from C4. Both the input channel and SVD components are sorted from small to large for better visualization. As shown in Figure 1 , we observe a highly sparse structure where an appropriate combination of input channels (green rectangle) and singular value components (yellow rectangle) can effectively approximate the full computation. Building on these, we propose R-Sparse , a simple yet effective framework that decompose the computation of each linear layer with a sparse and low-rank components. For the sparse portion, our approach identifies sparse channels by selecting those with large magnitude values and loads only the corresponding rows of weights into SRAM for computation. For the low-rank components, we route the non-sparse channels to a low-rank modules that obtained from an offline low-rank decomposition of the original weights. R-Sparse can be applied to both attention and MLP modules that achieves higher sparsity levels. Additionally, we find the patterns of sparse and low-rank combinations vary across different layers. With

Read the complete text