RSQ : Learning from Important Tokens Leads to Better Quantized LLMs
Yi-Lin Sung, Prateek Yadav, Jialu Li, Jaehong Yoon, Mohit Bansal · 2025
Read RSQ : Learning from Important Tokens Leads to Better Quantized LLMs by Yi-Lin Sung, Prateek Yadav, Jialu Li, Jaehong Yoon, Mohit Bansal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01820.txt.
Introduction
Yi-Lin Sung and co-authors shrink large language models by teaching the compression which tokens matter. Their 2025 paper improves layer-wise quantization, which rounds a model's numbers down to fewer bits without retraining. Earlier methods treated every output token equally when fitting each layer.
Their recipe, RSQ, first rotates the model to tame outlier values, then scales each token's features by its importance, measured by attention or dynamic cues, and only then quantizes. Important tokens thus dominate the fit. The finding is intuitive once stated: a compressed model should spend its limited precision where the meaning concentrates, not spread it evenly over filler.
Catalog record
| CREATOR | Yi-Lin Sung, Prateek Yadav, Jialu Li, Jaehong Yoon, Mohit Bansal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01820
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RSQ : Learning from Important Tokens Leads to Better Quantized LLMs Yi-Lin Sung, Prateek Yadav, Jialu Li, Jaehong Yoon, Mohit Bansal Abstract Layer-wise quantization is a key technique for efficiently compressing large models without expensive retraining. Previous methods typically quantize the weights of each layer by “uniformly” optimizing the layer reconstruction loss across all output tokens. However, in this paper, we demonstrate that better-quantized models can be obtained by prioritizing learning from important tokens ( e.g. which have large attention scores). Building on this finding, we propose RSQ (Rotate, Scale, then Quantize), which (1) applies rotations (orthogonal transformation) to the model to mitigate outliers (those with exceptionally large magnitude), (2) scales the token feature based on its importance, and (3) quantizes the model using the GPTQ framework with the second-order statistics computed by scaled tokens. To compute token importance, we explore both heuristic and dynamic strategies. Based on a thorough analysis of all approaches, we adopt attention concentration, which uses attention scores of each token as its importance, as the best approach. We demonstrate that RSQ consistently outperforms baseline methods across multiple downstream tasks and three model families: LLaMA3, Mistral, and Qwen2.5. Additionally, models quantized with RSQ achieve superior performance on long-context tasks, further highlighting its effectiveness. Lastly, RSQ demonstrates generalizability across various setups, including different model sizes, calibration datasets, bit precisions, and quantization methods. Our code is available at https://github.com/ylsung/rsq . 1 Introduction Large language models (LLMs) ( Georgiev et al., 2024 ; Achiam et al., 2023 ) have recently achieved great success and have transformed the landscape of artificial intelligence. However, the substantial computational demands associated with these models pose significant challenges to their usage and deployment, especially in resource-constrained scenarios. Weight quantization ( Han et al., 2016 ; Wu et al., 2016 ) is a widely used technique for reducing the computational costs of LLMs by representing weight values with fewer bits. Among various approaches, post-training quantization (PTQ) ( Frantar & Alistarh, 2022 ; Liu et al., 2021 ) is particularly favored by practitioners, as it enables the quantization of pre-trained LLMs using only a small calibration dataset, eliminating the need for expensive retraining. We focus on the layer-wise post-training quantization scheme ( Hubara et al., 2020 ; Li et al., 2021 ; Frantar et al., 2023 ) that has been demonstrated to be both effective and efficient for quantizing large models. Layer-wise quantization methods quantize an LLM’s weights one layer at a time by minimizing the token-level feature distance between the outputs of the original and quantized weights (i.e., the layer reconstruction loss, \lVert\mathbf{W}\mathbf{X}-\tilde{\mathbf{W}}\mathbf{X}\rVert^{2}_{2} ). Several advancements have been made to improve layer-wise quantization techniques in the past few years. For example, GPTQ ( Frantar et al., 2023 ) improves the efficiency and stability to compute second-order statistics and their inverses. QuIP# ( Tseng et al., 2024 ) and AQLM ( Egiazarian et al., 2024 ) represent quantized weights with vectors rather than fixed scalars. Additionally, QuIP ( Chee et al., 2023 ) and QuaRot ( Ashkboos et al., 2024b ) demonstrate through empirical studies that weight outliers–parameters with unusually large magnitudes–can be effectively mitigated by applying orthogonal transformations. Previous methods commonly perform layer-wise weight quantization by optimizing the layer reconstruction loss across all input tokens uniformly. However, research has shown that LLMs do not treat all tokens equally: (1) StreamingLLM ( Xiao et al., 2024 ) shows that initial tokens often have strong attention scores, (2) H 2 O ( Zhang et al., 2023 ) reveals that some tokens in KV cache contribute most of the attention values while decoding, and (3) RHO-1 ( Lin et al., 2024c ) demonstrates not all tokens are equal in training LLMs. Since quantized models inherently lose information due to the reduced capacity of lower bit representations, we argue that it should be particularly crucial for them to focus on learning and preserving the most critical information during the quantization process to maximize their performance. Inspired by these insights, we reconsider the conventional approach in quantization methods by optimizing the layer reconstruction loss over only a subset of important input tokens ( i.e. , using only the first 1/4 of the tokens). Our findings reveal that this strategy improves the quantized model’s accuracy across ten downstream tasks by up to 2.2%. Building on our findings and previous approaches, we propose RSQ to quantize the model in three steps: (1) rotate (orthogonally transform) the model to mitigate weight outliers, (2) scale the token feature based on its importance, and (3) quantize the weights using the GPTQ mechanism while leveraging token importance. We note that the token importance integrates seamlessly into the GPTQ framework in the third step, ensuring both compatibility and efficiency. Fig. 1 illustrates the three steps in RSQ . In this paper, we explore two categories of approaches for obtaining token importance: (1) heuristic approaches and (2) dynamic approaches. Within the heuristic category, we investigate methods such as First-N and First&Last-N, which prioritize initial tokens and a combination of initial and final tokens for quantization, respectively. These approaches outperform the conventional quantization method of optimizing across all tokens, achieving peak performance when N is roughly 5–10% of the total tokens. It is important to note that the initial or final tokens do not inherently contain more meaningful semantic information. Instead, their importance likely stems from their positional characteristics and their tendency to receive stronger attention scores ( Xiao et al., 2024 ; Sun et al., 2024a ) . In this aforementioned approach, token importance was determined solely based on heuristics (e.g., token positions). To further improve performance beyond heuristic methods, we also explore several dynamic approaches for computing token importance individually based on each input. Specifically, we investigate TokenFreq, where less frequent tokens are considered more important; ActNorm, which prioritizes tokens with larger norms ; ActDiff, where tokens whose features change less after one layer are assigned higher importance; TokenSim, which gives greater weight to tokens that are less similar to others; and AttnCon, where tokens receiving higher attention scores are considered more important. Among these, AttnCon performs the best as it more explicitly models each token’s impact on the other tokens. Therefore, we adopt it as our final strategy. In these dynamic approaches, we also observe the presence of positional biases, similar to those seen in heuristic methods. To ensure that tokens in “less important” positions are not wasted, we introduce a data augmentation strategy. This involves expanding each sample by shifting it forward by several positions and adding the shifted samples to the calibration dataset. This augmentation ensures a more comprehensive utilization of these tokens. We evaluate RSQ on WikiText-2 and a diverse set of tasks, including LAMBADA, WinoGrande, ARC, HellaSwag, PIQA, MMLU, GSM8k, and TruthfulQA, using three models: LLaMA3-3B-Instruct, Mistral-Nemo-12B, and Qwen2.5-7B. Our results demonstrate that RSQ achieves absolute improvements of 1.6%, 0.9%, and 0.4% in average accuracy across these tasks for the three models, respectively. To facilitate a comprehensive evaluation, we compare RSQ against baseline methods on subsets of various long-context benchmarks, including LongEval, L-Eval, LongICLBench, and LongCodeArena. Our results show that RSQ achieves improvements of 3.0%, 2.5%, 0.6%, and 0.025, respectively, over QuaRot on these benchmarks. Lastly, we demonstrate the generalizability of RSQ