Accelerating Transformer Inference and Training with 2:4 Activation Sparsity
Daniel Haziza, Timothy Chou, Dhruv Choudhary, Luca Wehrstedt, Francisco Massa, Jiecao Yu, Geonhwa Jeong, Supriya Rao, Patrick Labatut, Jesse Cai · 2025
Read Accelerating Transformer Inference and Training with 2:4 Activation Sparsity by Daniel Haziza, Timothy Chou, Dhruv Choudhary, Luca Wehrstedt, Francisco Massa, Jiecao Yu, Geonhwa Jeong, Supriya Rao, Patrick Labatut, Jesse Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16672.txt.
Introduction
Daniel Haziza and co-authors speed up transformer models by exploiting a structured sparsity pattern that graphics hardware already loves. The 2:4 pattern keeps exactly two of every four values, and GPUs can race through such data. The trick is applying it to activations, the live signals inside the network, without losing accuracy.
The team leans on a quirk of Squared-ReLU activations, which are naturally sparse in just this way. Feed-forward layers then run faster in both training and inference, in forward and backward passes alike. No accuracy tax is paid for the gain. As models grow, free speed from structure already inside them is the cheapest kind there is.
Catalog record
| CREATOR | Daniel Haziza, Timothy Chou, Dhruv Choudhary, Luca Wehrstedt, Francisco Massa, Jiecao Yu, Geonhwa Jeong, Supriya Rao, Patrick Labatut, Jesse Cai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 12 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-16672
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Accelerating Transformer Inference and Training with 2:4 Activation Sparsity Daniel Haziza, Timothy Chou, Dhruv Choudhary, Luca Wehrstedt, Francisco Massa, Jiecao Yu, Geonhwa Jeong, Supriya Rao, Patrick Labatut, Jesse Cai Abstract In this paper, we demonstrate how to leverage 2:4 sparsity, a popular hardware-accelerated GPU sparsity pattern, to activations to accelerate large language model training and inference. Crucially we exploit the intrinsic sparsity found in Squared-ReLU activations to provide this acceleration with no accuracy loss . Our approach achieves up to 1.3x faster Feed Forward Network (FFNs) in both the forwards and backwards pass. This work highlights the potential for sparsity to play a key role in accelerating large language model training and inference. 1 Introduction & Related Work The rapid growth of Large Language Models (LLMs) in recent years has been driven by a corresponding surge in GPU FLOPs. While some of this gain has come from hardware improvements, a large portion has come from reducing operator precision from 32\rightarrow 16\rightarrow 8 bits. However, as we approach the 0 bit quantization asymptote, researchers have turned to alternative methods to reduce LLM compute. One promising approach is sparsity, which seeks to avoid unnecessary computation to accelerate the model. Existing work on sparsity for LLMs has primarily focused on inference. SparseGPT ( Frantar & Alistarh, 2023 ) and others have shown that existing dense LLM weights can be made sparse, either through post training pruning ( Sun et al., 2023 ; Zhang et al., 2024a ) or fine-tuning ( Kurtic et al., 2023 ) , enabling subsequent inference acceleration. Sparsity has also been applied to activations ( Liu et al., 2024b ; Lee et al., 2024 ; Zhang et al., 2024b ) to speed up batch size=1 decode inference workloads via selective weight loading. However, these approaches are unable to provide speedups during LLM training or prefill, where we experience compute-bound workloads. Mozaffari et al. (2024) , Hu et al. (2024) were able to accelerate LLM training with 2:4 sparsity, but applied to the model weights. Moreover, in all of these works, the final sparse model comes with a small reduction in model accuracy compared to the dense model. In this paper, we investigate the application of 2:4 sparsity to accelerate both LLM training and inference by sparsifying model activations. Crucially, we exploit the intrinsic sparsity found in certain activation functions, which is not present in the dense model weights. Our main contributions are as follows: (1) A recipe to accelerate Squared-RELU LLM training and inference with 2:4 sparsity. (2) Accuracy experiments showing minimal performance degradation when using our emulated recipe compared to the dense baseline. (3) Kernel benchmarks showing up to 1.3x faster FFNs when using 2:4 sparsity. Squared ReLU: Recently, SwiGLU = (Swish_{\beta}(XW_{1})\odot XW_{3})W_{2} ( Shazeer, 2020 ) has been used as the activation function in the Feed-Forward Network blocks (FFN) of multiple LLMs ( Touvron et al., 2023 ) ( Liu et al., 2024a ) . However, ( So et al., 2021 ) suggests that Squared-ReLU = ({\max(0,XW_{1})}^{2})W_{2} is at least as good as SwiGLU, which we confirmed in our experiments (Table 1 ). Larger models have since been trained successfully with Squared-ReLU ( Adler et al., 2024 ) . Figure 1: We plot the sparsity level progression of different FFN layers over a training run. Replacing SwiGLU with Squared-ReLU does not impact the model accuracy (See Table 1 ), and makes the FFN activations highly sparse during training & inference (84-98% sparse for this model). While the model accuracy remains similar, switching the activation function to Squared-ReLU has an interesting side-effect: activation sparsity naturally emerges during training. Intuitively, this makes sense as Swish_{\beta}(x)=\frac{x}{1+e^{-\beta x}} will be 0 if and only if x=0 whereas ReLU^{2}={\max(0,x)}^{2} maps any x<0 to 0. In theory, with normal centered inputs, we expect to see 50% sparsity in the activations - which is what we observe at model initialization. In practice, however, we observe that the sparsity level rapidly increases during training, reaching 85-98% depending on the layer (Figure 1 ). While we do not offer an explanation for the emergence of this phenomenon, this means that a Squared ReLU model spends a large amount of GPU time and energy multiplying zeros. 1 1 1 While we only examine Squared-RELU models in this paper, this approach can be adapted to any activation function that achieves similar sparsity levels. Figure 2: Given an activation tensor A of shape [seqlen,features] (Figure 2a), we can accelerate the computation of AB if A is 2:4-sparse token-wise (Figure 2b), or A^{T}B if A is 2:4-sparse feature-wise (Figure 2c). To respect the 2:4 sparsity constraint, it is possible that some values need to be dropped when the tensor A is sparsified (in red). The more sparse the original tensor is, the less likely we are to drop values - if no values are dropped, the calculation is exact. Hardware-accelerated (2:4) sparsity: To accelerate this model, we leverage sparsity. Modern GPUs have specialized units for matrix multiplication (TensorCores), and these units can only accelerate a single sparsity format: 2:4 sparsity ( Mishra et al., 2021 ) . A tensor is 2:4 sparse if for every 4 consecutive values, at most 2 are non-zeros (Figure 2 ). If a tensor A is 2:4-sparse (50% sparsity), it is theoretically possible to compute AB 2x faster, because we skip every other multiplication - in practice we see 1.5x-1.7x speedups for 2:4 FP8 matrix multiplication on H100 compared to dense FP8 depending on the matrix shape. 2 Approach Inspired by Jeong et al. (2024) , we aim to approximate the second 2 2 2 Note that the first linear layer cannot be accelerated as its inputs are not sparse. Further savings could be achieved if the residual can be made sparse. matrix multiply of the FFN, Y_{2}W_{2} , with a 2:4 sparse matrix multiply, as Y_{2}=ReLU^{2}(X_{1}W_{1}) and is mostly zero. If more than 2 elements are non-zeros in a block of 4 in Y_{2} , we naively keep the 2 highest magnitude scalars and drop the others. In practice, \sim 1% of non-zero values are dropped, but this can vary depending on the sparsity level. Figure 3: Pseudo-code for our proposed replacement FP8 Squared-ReLU FFN. We implemented 2 new kernels, so that the entire FFN forward pass can be implemented with only 3 kernels: the existing FP8 matmul for the first linear layer, a new kernel that applies the activation, the sparsification, and FP8 quantization, and another new 2:4 GEMM kernel that supports FP8 rowwise scaling. To sparsify the activations we adapt the fast sparsification routines outlined in Cai et al. () 3 3 3 These routines have been open sourced in torchao ( torchao maintainers & contributors, 2024 ) Notably, this also provides inference speedups in the compute bound regime (Figure 5 ), such as during prefilling, with speculative decoding, or higher batch sizes. This is different from existing activation sparsity work ( Liu et al., 2024b ; Lee et al., 2024 ) that aims at reducing the amount of weights loaded from memory, which can only accelerate batch size=1 decode. Figure 4: The compute graph for a Squared-ReLU FFN during training. As ReLU^{2}(y_{1}) is highly sparse, any matrix multiplication involving y_{2} or dy_{1}