From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation

Yuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang · 2026

Read From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation by Yuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18865.txt.

Introduction

Yuxin Ren and three colleagues replace attention with simpler sequential modules by following sparsity. Self-attention costs grow with the square of token count, yet naive replacement loses accuracy, above all at larger scales. The authors find layers differ in sparsity, so some layer jobs can be swapped for far simpler modules without loss. Layer-wise distillation tests on vision transformers show sparser layers swap with smaller drops, and guided sparsity shrinks the student-teacher gap. The result cuts size and latency for attention replacement.

Catalog record

CREATORYuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50356
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18865

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Sparsity to Simplicity: Enabling Simpler Sequential Replacements via Sparse Attention Distillation Yuxin Ren, Maxwell D Collins, Miao Hu, Huanrui Yang Abstract Self-attention serves as the core foundation of large-scale transformer pretraining, but its quadratic token interaction cost makes inference expensive. Replacing attention with simpler sequential modules is appealing, yet naïve substitution is often lossy, especially at larger scales. This paper revisits attention replacement through the lens of sparsity. Based on the observation of diverse sparsity patterns across transformer layers, we posit that pretrained transformers decompose the complex token dependency across tokens into various sequence-to-sequence mappings of diverse complexities, where some layer functionalities can be approximated and replaced with much simpler sequential modules without loss. We evaluate this premise using a plug-and-play layer-wise distillation framework to approximate and replace attention functionalities in pretrained vision transformer models. Controlled group-wise replacements under a fixed training budget reveal a clear pattern: substituting layers with sparser attention incurs substantially smaller accuracy drops than replacing denser ones. We further impose explicit attention sparsity on the pretrained ViT via AViT-style token retention and perform sparsity-guided distillation for sequential replacing models, where we see increasing teacher sparsity consistently reduces the student–teacher gap. The proposed method achieves efficient attention replacement for reduced parameter size and latency through the guidance of attention sparsity. Code is available at https://github.com/aliothren/FAR . 1 Introduction Transformers have become the dominant backbone across vision, language, and multimodal learning [ 27 , 5 , 6 , 18 ] . Their success is largely attributed to the self-attention mechanism, which performs flexible token mixing through all-pair interactions, providing a powerful method for modeling long-range dependencies and aggregating context adaptively [ 3 ] . However, attention comes with quadratic complexity in sequence length and heavy activation-to-activation multiplications, resulting in high latency and memory overhead during inference [ 23 , 4 ] . This tension has motivated extensive research on efficient attention. Most efficiency-oriented approaches modify the attention computation itself through linearized or recurrent formulations [ 13 , 8 ] to reduce the cost while maintaining accuracy, yet such simplifications inevitably reduce the expressive capacity of attention. Recent theoretical work attributes this gap to the lossy hidden-state memory of recurrent models, which prevents the exact retrieval and combination of distant contextual information [ 30 ] . This signifies why attention is indispensable during large-scale pretraining, as it uniquely supports lossless retrieval of token relationships across arbitrary context lengths. Nevertheless, the same level of expressivity may not be required during inference, where attention operates on already contextualized representations produced by preceding layers, and its computation may be effectively far less complex and more structured than its dense all-pair form suggests. Empirically, pretrained transformers exhibit substantial redundancy in attention at inference time: attention maps are often sparse and low-rank, admitting compressed structure even without explicit regularization [ 9 , 1 ] . Crucially, this sparsity is not uniform across the network. Different layers show markedly different token-interaction patterns, indicating that the role of attention varies across depth and that certain blocks may rely on only a small subset of token interactions. These observations motivate a shift in how we think about attention replacement. Rather than asking whether a simple module can match attention everywhere, we ask when and where attention behaves like a mapping that can be replaced with minimal loss. This leads to the central hypothesis of this paper. Hypothesis. Pretraining decomposes complex dependencies into block-wise mappings with different effective complexity, and inference-time attention sparsity is a signature of this complexity. We therefore expect sparse blocks to be more replaceable by simpler sequential modules, and explicit sparsification of the teacher to further improve distillation. We refer to this sparsity guided replacement principle as Sparsity to Simplicity (S2S) . S2S couples a unified replacement scheme with sparsity aware supervision, using attention sparsity to identify replaceable blocks and to distill attention behavior into simpler sequential substitutes. We evaluate the hypothesis in a series of controlled experiments. We start by designing a unified, plug-and-play layer-wise attention distillation and replacement scheme. The scheme replaces attention blocks with sequential modules such as Mamba [ 8 ] and LSTMs [ 11 ] that match the original input and output interface, so the backbone structure remains unchanged and pretrained weights can be reused as much as possible. This scheme provides a unified baseline of the feasibility of attention replacement across layers, backbones, and substitute model, where we use the performance gap before and after replacement to reflect how well a sequential module can approximate the target attention mapping. Guided by this, we test the link between attention sparsity and replacement feasibility across the model depth. We introduce an AViT-style token retention [ 32 ] as an explicit and controllable attention sparsification mechanism, which aligns with the natural depth-varying attention sparsity trend observed in pretrained ViT models. We replace attention blocks at different layer groups with sequential modules under identical training conditions to isolate the approximation difficulty of the replaced mapping. Replacements in sparser blocks consistently incur smaller accuracy drops and lower distillation error than replacements in denser blocks, supporting the view that sparsity marks simpler mappings that are easier to approximate. Further experiments on explicitly controlling the attention sparsity level also shows that increasing teacher attention sparsity consistently shrinks the student-teacher gap, suggesting that explicit attention sparsification exposes a desirable behavior that is easier for alternative sequential modules to learn. Following the S2S principle, we demonstrate that replacing the most replaceable 3 late-layer attentions in pretrained DeiT with designed BiMamba blocks under matched A-ViT sparsity preserves sparse-attention accuracy while improving efficiency (up to 1.71 \times estimated speedup at 384^{2} ). The effectiveness of S2S also extends to DeiT transfer benchmarks, where the replacement models can be finetuned with no clear performance gap to their attention-based counterparts. 2 Related Work Sparsity, Redundancy and Efficiency in Attention. A body of work has revealed substantial redundancy in pretrained attention at inference time. Bhojanapalli et al. [ 1 ] and He et al. [ 9 ] show that only a small subset of heads and token interactions contribute meaningfully, while many attention weights can be pruned with limited accuracy loss. Dynamic token pruning methods such as DynamicViT [ 19 ] , TokenLearner [ 20 ] , and A-ViT [ 32 ] exploit this redundancy by adaptively reducing token computation. In language models, a related line of work improves long-context decoding by making inference more selective through memory and KV-cache management [ 31 , 33 , 2 , 16 ] . These methods expose that inference behavior is often dominated by a small set of important tokens and interactions, but they still retain attention as the underlying token-mixing operator. In contrast, we use layer-wise sparsity as a signal of where attention mappings become easier to approximate, and we study when such sparse behavior can be distilled into simpler sequential modules that replace attention itself. Efficient Architectures Beyond Self-Attention. The quadratic cost of self-attention has motivated attention-free or attention-reduced architectures that replace all-pair token mixing with simpler mechanisms. Examples include MLP-Mixer [ 24 ] and related MLP-based mixers, as well as recurrent or state-space alternatives such as RetNet [ 21 ] and Mamba [ 8 ] . While these designs improve scalability,

Read the complete text