Spark Transformer: Reactivating Sparsity in FFN and Attention
Chong You 1, Kan Wu 2, Zhipeng Jia 2, Lin Chen 1, Srinadh Bhojanapalli 1, Jiaxian Guo 1, Utku Evci 3, Jan Wassenberg 3, Praneeth Netrapalli 3, Jeremiah J. Willcock 3, Suvinay Subramanian 2, Felix Chern 1, Alek Andreev 3, Shreya Pathak 3, Felix Yu 1, Prateek Jain 3, David E. Culler 2, Henry M. Levy 2 · 2025
Read Spark Transformer: Reactivating Sparsity in FFN and Attention by Chong You 1, Kan Wu 2, Zhipeng Jia 2, Lin Chen 1, Srinadh Bhojanapalli 1, Jiaxian Guo 1, Utku Evci 3, Jan Wassenberg 3, Praneeth Netrapalli 3, Jeremiah J. Willcock 3, Suvinay Subramanian 2, Felix Chern 1, Alek Andreev 3, Shreya Pathak 3, Felix Yu 1, Prateek Jain 3, David E. Culler 2, Henry M. Levy 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06644.txt.
Introduction
Chong You and co-authors present the Spark Transformer, a neural network design that wakes up the sparsity sleeping inside large language models, in a 2025 paper. In trained networks of this kind, most inner units stay silent for any given word, a fact called the lazy neuron effect. Older designs gained speed from that silence, but newer ones dropped the switch that caused it, and past attempts to restore it hurt quality or slowed training.
Spark keeps only the top few units active in each layer and in the attention step that weighs context. A fast statistical shortcut picks those units without slow sorting, and spare parts of the network are reused as a cheap scout that predicts which units will fire. The result is a model with sparsity under direct control that keeps its quality, size, and normal training routine while doing less work per word.
Catalog record
| CREATOR | Chong You 1, Kan Wu 2, Zhipeng Jia 2, Lin Chen 1, Srinadh Bhojanapalli 1, Jiaxian Guo 1, Utku Evci 3, Jan Wassenberg 3, Praneeth Netrapalli 3, Jeremiah J. Willcock 3, Suvinay Subramanian 2, Felix Chern 1, Alek Andreev 3, Shreya Pathak 3, Felix Yu 1, Prateek Jain 3, David E. Culler 2, Henry M. Levy 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06644
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Spark Transformer: Reactivating Sparsity in FFN and Attention Chong You 1, Kan Wu 2, Zhipeng Jia 2, Lin Chen 1, Srinadh Bhojanapalli 1, Jiaxian Guo 1, Utku Evci 3, Jan Wassenberg 3, Praneeth Netrapalli 3, Jeremiah J. Willcock 3, Suvinay Subramanian 2, Felix Chern 1, Alek Andreev 3, Shreya Pathak 3, Felix Yu 1, Prateek Jain 3, David E. Culler 2, Henry M. Levy 2, Sanjiv Kumar 1 Abstract The discovery of the lazy neuron phenomenon [ 49 ] in trained Transformers, where the vast majority of neurons in their feed-forward networks (FFN) are inactive for each token, has spurred tremendous interests in activation sparsity for enhancing large model efficiency. While notable progress has been made in translating such sparsity to wall-time benefits across CPUs, GPUs, and TPUs, modern Transformers have moved away from the ReLU activation function crucial to this phenomenon. Existing efforts on re-introducing activation sparsity, e.g., by reverting to ReLU, applying top- k k masking, or introducing a sparse predictor, often degrade model quality, increase parameter count, complicate or slow down training. Sparse attention , the application of sparse activation to the attention mechanism, often face similar challenges. This paper introduces the Spark Transformer , a novel architecture that achieves a high level of activation sparsity in both FFN and the attention mechanism while maintaining model quality, parameter count, and standard training procedures. Our method realizes sparsity via top- k k masking for explicit control over sparsity level. Crucially, we introduce statistical top- k k , a hardware-accelerator-friendly, linear-time approximate algorithm that avoids costly sorting and mitigates significant training slowdown from standard top- k k operators. Furthermore, Spark Transformer reallocates existing FFN parameters and attention key embeddings to form a low-cost predictor for identifying activated entries. This design not only mitigates quality loss from enforced sparsity, but also enhances wall-time benefit. Pretrained with the Gemma-2 recipe, Spark Transformer demonstrates competitive performance on standard benchmarks while exhibiting significant sparsity: only 8% of FFN neurons are activated, and each token attends to a maximum of 256 tokens. This sparsity translates to a 2.5 × \times reduction in FLOPs, leading to decoding wall-time speedups of up to 1.79 × \times on CPU and 1.40 × \times on GPU. 1 Introduction The machine learning field has experienced a rapid expansion in the scale of Transformer models [ 4 , 3 , 24 , 1 ] , significantly advancing the state-of-the-art in language understanding and generation. However, this pursuit of larger models is often constrained not by inherent limitations in model quality [ 40 ] , but by the soaring computational costs [ 71 , 63 ] associated with the number of parameters. This challenge is compounded by the trend towards models capable of processing increasingly longer input sequences [ 67 ] , where computational demands scale proportionally with input length. (a) FLOPs per token vs. quality ( 1/6 of full training) (b) Downstream evaluation Prefill speed (ms/token) Decode speed (ms/token) Gemma-2 28 141 Spark Transformer 15 86 (c) CPU inference wall time Figure 1 : Spark Transformer improves inference efficiency via high-level activation sparsity in both FFN and attention, while being nearly quality neutral. (a) Comparison to prior work in terms of relative FLOPs per token at 8k sequence length (y-axis) vs relative training loss (x-axis) 2 2 2 All models in Figure 1(a) were trained with standard Gemma-2 recipe for 1/6 of the full pretraining iterations. All other results in this paper are obtained for a fully pretrained Spark Transformer. . [ \blacksquare ] We use standard Gemma-2 [ 27 ] as baseline, which has no activation sparsity. [ • ] Methods employing activation sparsity within the FFN layers only. Our Spark FFN achieves the most favorable trade-off compared to ReLU, ReLU 2 , and Topk , which refer to standard Gemma-2 with activation function switched to ReLU [ 59 ] , ReLU 2 [ 86 ] , and the composition of Topk and GELU, respectively. [ \blacktriangle ] Combining Spark FFN (with 8% activated parameters) with Spark Attention (with at most 256 attended tokens), our Spark Transformer achieves performance comparable to Gemma-2 while reducing FLOPs to 40%. (b) Evaluation on standard downstream tasks confirms near-quality neutrality of Spark Transformer. (c) Prefill / decode wall time demonstrate a 1.86 \times /1.64 \times speedup resulting from FLOPs reduction. Results are obtained on a 4-Core CPUs for prompts of 4096 tokens. For prefill, the prompt is chunked into batches of 64 tokens, following a default setup of gemma.cpp [ 30 ] . Activation sparsity has emerged as a prominent technique for mitigating the computational burdens associated with both large model sizes and long input sequences. For large models, activation sparsity reduces computational cost by activating only a small fraction of the model’s parameters for each input. This approach has attracted considerable attention due to the observed lazy neuron phenomenon [ 49 ] where natural activation sparsity occurs in the feed-forward networks (FFNs) of traditional Transformer models like T5 [ 66 ] and ViT [ 23 ] , without explicit enforcement [ 85 ] . Subsequent work has successfully demonstrated wall-time benefits from activation sparsity on multiple hardware platforms, including CPU [ 84 ] , GPU [ 77 , 53 ] , and TPU [ 83 ] . Despite the great success, a fundamental challenge arises in the application of sparse activation to the latest and state-of-the-art models. That is, the inherent sparsity, which is pivotal for obtaining efficiency, is largely absent due to the adoption of gated non-ReLU activation functions [ 18 ] , widely adopted in, e.g., Mistral [ 37 ] , Gemma [ 27 ] , LLAMA [ 24 ] . This motivates the following question of reactivating sparsity : Can we re-introduce a high level of activation sparsity in recent Transformers, without hurting their quality? To answer this question, prior work has explored reverting to ReLU and its variants [ 59 , 86 ] , incorporating top- k thresholding [ 83 , 76 , 78 ] , or introducing a low-cost sparsity predictor [ 53 , 84 , 77 , 83 ] . However, (Challenge #1) these methods often lead to a degradation in model quality (see Figure 1(a) ) . Furthermore, (Challenge #2) top- k thresholding based methods involve sorting to obtain maximally activated neurons, which is inefficient on ML accelerators, such as TPUs [ 39 ] , with possibly a 10\times training slowdown (see Figure 6 ) . Finally, (Challenge #3) the inclusion of a sparsity predictor often increases training pipeline complexity and incurs additional training costs and parameters. Sparse attention, the application of activation sparsity to the attention mechanism, presents similar challenges. A common strategy is top- k attention [ 31 ] , which applies a top- k mask to the attention coefficients. This can be augmented with a low-cost predictor to further enhance efficiency [ 69 , 82 , 46 ] . Nevertheless, achieving both high sparsity and accurate prediction without resorting to complex procedures, training slowdown from top- k , and compromising quality remains an open problem. Contributions. This paper introduces the Spark Transformer , a novel architecture that achieves a strong level of activation sparsity in both FFN and attention mechanism with minimal