Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

Rahul Raman, Khushi Sharma, Sai Qian Zhang · 2025

Read Rethinking the Outlier Distribution in Large Language Models: An In-depth Study by Rahul Raman, Khushi Sharma, Sai Qian Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21670.txt.

Introduction

Rahul Raman, Khushi Sharma, and Sai Qian Zhang dig into why large language models grow outlier values that hurt shrinking for small devices. Big spikes in a few spots cause large errors when models are packed into fewer bits. Two spike kinds are known, huge acts and fixed channel spikes.

They trace how each kind forms through model depth and training. That root view lets them craft cheap fixes. Their fixes clear most spikes of both kinds with tiny loss in skill. Smoother models pack better for phones and edge gear, a key step for wide use of these tools.

Catalog record

CREATORRahul Raman, Khushi Sharma, Sai Qian Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21670

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking the Outlier Distribution in Large Language Models: An In-depth Study Rahul Raman, Khushi Sharma, Sai Qian Zhang Abstract Investigating outliers in large language models (LLMs) is crucial due to their significant impact on various aspects of LLM performance, including quantization and compression. Outliers often cause considerable quantization errors, leading to degraded model performance. Identifying and addressing these outliers can enhance the accuracy and efficiency of the quantization process, enabling smoother deployment on edge devices or specialized hardware. Recent studies have identified two common types of outliers in LLMs: massive activations and channel-wise outliers. While numerous quantization algorithms have been proposed to mitigate their effects and maintain satisfactory accuracy, few have thoroughly explored the root causes of these outliers in depth. In this paper, we conduct a comprehensive investigation into the formation mechanisms of these outliers and propose potential strategies to mitigate their occurrence. Ultimately, we introduce some efficient approaches to eliminate most massive activations and channel-wise outliers with minimal impact on accuracy. 1 Introduction Large Language Models (LLMs) have emerged as a cornerstone in the field of natural language processing (NLP), transforming how we approach various linguistic tasks. These models, with their ability to understand and generate human-like text, have revolutionized applications ranging from conventional NLP tasks such as machine translation Huang et al. (2023) ; Xu et al. (2024) ; Zhu et al. (2023) , sentiment analysis Miah et al. (2024) ; Wang et al. (2024) ; Deng et al. (2023) to advanced tasks such as code generation Kazemitabaar et al. (2023) ; Thakur et al. (2024) ; Nakkab et al. (2024) . However, the enormous size of LLMs, often reaching billions of parameters, presents substantial challenges for deployment, necessitating the use of techniques that enable efficient inference. To address this, Post-Training Quantization (PTQ) Frantar et al. (2022) ; Xiao et al. (2023) ; Lin et al. (2024a) ; Yao et al. (2022) provides a practical, low-cost approach for model quantization, either completely training-free or with minimal calibration effort Cai et al. (2020) ; Li et al. (2021) . In comparison to Quantization-Aware Training (QAT), which demands multiple fine-tuning iterations, PTQ incurs much lower computational costs, making it suitable for LLM. Unfortunately, outliers in LLM activations and KV vectors Dettmers et al. (2022) ; Zeng et al. (2022) introduce significant magnitude variations among LLM elements, which in turn lead to a notable drop in model accuracy when low-precision PTQ is applied Xiao et al. (2023) ; Tseng et al. (2024) ; Ashkboos et al. (2024b) . Prior research has identified two types of outliers in LLM activations. The first, massive activations (MAs), commonly appear across various LLMs and are typically linked to specific tokens in certain channels Sun et al. (2024) . The second type, channel-wise outliers Dettmers et al. (2022) ; Xiao et al. (2023) ; Ashkboos et al. (2024b) , manifests in bulk within specific channels. These findings have inspired a two-stage approach in modern quantization techniques: initially, methods are employed to eliminate outliers in the pretrained LLM, resulting in a model with a smoother value distribution in its activations. Subsequently, quantization algorithms such as GPTQ Frantar et al. (2022) and OBQ Frantar and Alistarh (2022) are applied to produce low-precision LLMs, as shown in Figure 1 . Figure 1: Given a pretrained LLM, techniques are first applied to smooth out the outliers in its activations. The resulting model is then quantized, achieving superior accuracy. Outlier smoothing is a crucial step in achieving efficient LLM quantization. Understanding the root causes of outliers is essential for developing effective quantization techniques and gaining deeper insights into model behavior and robustness. While prior studies have identified the presence of MAs and channel-wise outliers, and proposed methods to mitigate them Sun et al. (2024) ; Liu et al. (2024) ; Bini et al. (2024) ; Xiong et al. (2024) , none have explored the fundamental reasons behind the existence of these outliers from a numerical perspective, particularly with operator-level granularity. This finer-grained understanding is crucial, as different layers and operators may contribute uniquely to the formation and propagation of LLM outliers, influencing both performance and accuracy in low-precision LLMs. Figure 2: (a) Architecture of a LLM decoder block. (b), (c) and (d) show the architectures of self-attention block, standard FFN (conventional MLP), and gated FFN (GLU), respectively. The notations will be used throughout the rest sections. In this work, we investigate the underlying reasons for the existence of outliers in LLMs at the operator level through extensive empirical analysis. Our study provides valuable insights to guide the development of effective outlier smoothing algorithms. Building on these findings, we propose some novel methods to efficiently mitigate the majority of massive activations and channel-wise outliers without compromising model accuracy. This significantly reduces the complexity of subsequent LLM quantization processes. In summary, our findings on LLM outliers can be summarized as follows: • We empirically demonstrate that massive activations (MAs) are predominantly generated in the initial layers the model. Once these MAs arise, they persist throughout the LLM, being propagated through subsequent layers via residual connections. • Previous studies indicate that the removal of MAs can significantly impact the quantization process. Surprising, our empirical analysis shows that eliminating MAs introduced by residual connections has no measurable effect on the model’s accuracy. Notably, these MAs constitute the majority of MAs in LLMs. • Channel-wise outliers in LLMs initially emerge due to the normalization operations within the model. The rescaling operation within the normalization layer exacerbates this issue by introducing an increasing number of channel-wise outliers. • Certain channels within the weight matrices can also contribute to the emergence of channel-wise outliers in the intermediate results of LLMs. 2 Background and Related Work 2.1 LLM Operations Modern LLMs (e.g., Llama series Touvron et al. (2023a) ; Touvron et al. (2023b) , GPT series Radford et al. (2019) ; Brown (2020) ) are constructed as a stack of transformer decoders, with each decoder comprising two fundamental components: a Self-Attention (SA) block and a feed forward network (FFN), as depicted in Figure 2 (a). During the LLM serving process, the input to the Self-Attention (SA) block is first processed by a normalization operation (e.g., LayerNorm or RMSNorm). As detailed in Figure 3 (d), this normalization consists of two key steps: standardization and rescaling. Specifically, the input X is normalized by subtracting its mean \mu_{X} and dividing by its standard deviation \sigma_{X} . Subsequently, each channel of the standardized output is scaled by a learnable parameter \gamma and shifted by another learnable parameter \beta . The output of the normalization operation is then multiplied with three weight matrices W_{Q} , W_{K} , and W_{V} , yielding the outputs referred to as query ( q ), key ( k ), and value ( v ), which is shown as x_{3} , x_{4} and x_{5} in Figure 2 , respectively. The resulting q and k , in combination with v , will then undergo multiplication, Softmax, and residual addition to generate the SA output, as shown in Figure 2 (b). The output from the SA will then be passed to the FFN for further processing, which typically involves a gated MLP Radford (2018) ; Radford et al. (2019) (Figure 2

Read the complete text