Vision Transformers Need Better Token Interaction
Su Linxiang · 2026
Read Vision Transformers Need Better Token Interaction by Su Linxiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-23868.txt.
Introduction
Linxiang Su revisits a quiet failure in vision transformers: patch tokens lose their local sharpness during long training even as whole image scores stay strong. Su argues the culprit is semantic diffusion, global meaning bleeding into patches beyond what local evidence warrants, not just high norm artifacts. Testing sparse attention as a light fix, swapping softmax for entmax-1.5 while keeping global links, preserves ImageNet probing and lifts segmentation markedly, with VOC mean IoU rising from 42.80 to 48.78.
Catalog record
| CREATOR | Su Linxiang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18823 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.23868
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision Transformers Need Better Token Interaction Su Linxiang Abstract Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize semantic diffusion : an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and [CLS] features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations. Figure 1: Visualization of the impact of our proposed sparse attention, compared with other baselines. We display the first PCA components of model outputs in RGB. 1 Introduction Vision Transformers (ViTs) have become a standard architecture for learning transferable visual representations ( 7 ; 2 ; 16 ) . Beyond strong image-level recognition performance, a desirable ViT representation should also preserve spatially meaningful patch features that transfer effectively to dense prediction tasks such as semantic segmentation and monocular depth estimation. However, recent studies suggest that these objectives can diverge during training. As training progresses, image-level performance may continue improving while dense representations gradually become noisier and less effective for downstream dense tasks. This phenomenon has been associated with artifacts or outlier tokens in ViTs ( 6 ; 20 ) . In this work, we argue that such artifacts are likely a symptom rather than the root cause. Instead, we hypothesize that the main issue is an optimization shortcut in which global semantic information diffuses into patch tokens beyond what is locally justified, a phenomenon we term semantic diffusion . LaSt-ViT attributes dense degradation to the combination of coarse image-level supervision and unrestricted global token interactions, which encourages background patches to participate in representing global semantics ( 20 ) . Other approaches attempt to mitigate this issue through register tokens, token specialization, attention modifications, or dense relational supervision ( 6 ; 14 ; 16 ; 21 ) . While effective, these methods do not directly address the shortcut itself. Register tokens mainly provide dedicated storage for leaked information, heuristic token filtering relies on manually designed criteria, and teacher-based dense constraints require an additional dense-good teacher model. Our starting point is that the objective should not be to eliminate global context or suppress background tokens altogether. Background regions often contain useful contextual cues, and globally shared semantics can still benefit downstream tasks. Instead, the goal should be to encourage useful token interactions : interactions that preserve local structure while selectively incorporating global contextual information. Motivated by this perspective, we study sparse attention as a simple and direct intervention on attention geometry. Rather than restricting receptive fields through local windows ( 13 ) or introducing additional register tokens ( 6 ) , we retain global connectivity but replace dense softmax attention with entmax-1.5 ( 17 ; 5 ) . This produces sparse and selective attention distributions through only a minimal architectural modification. 2 Related Work Artifacts in ViTs. ViT patch features can exhibit high-norm outliers, noisy PCA visualizations, or degraded dense-task performance despite strong image-level recognition ability. DINOv2 popularized register tokens as a practical mechanism for improving dense representations by introducing additional storage slots ( 6 ; 16 ) . Recent work further suggests that such artifacts may emerge systematically from attention dynamics rather than random optimization failures. In large language models, systematic outliers have been linked to the softmax attention mechanism itself, where they act as implicit context-aware scaling factors ( 1 ) . In vision transformers, LAST-ViT attributes artifacts to a lazy aggregation behavior, where semantically irrelevant background patches are exploited as shortcuts for representing global semantics ( 20 ) . Our work is aligned with these perspectives. However, instead of managing artifacts after they emerge, we focus on modifying the token interaction mechanism itself to make semantic diffusion less favorable during optimization. Dense Performance Degradation. Recent studies observe that prolonged self-supervised training can continuously improve global recognition performance while simultaneously degrading dense representations. This phenomenon was first implicitly observed in DINOv2 ( 16 ) and later studied more explicitly in DINOv3 ( 21 ) . DINOv3 proposes Gram anchoring, where an earlier checkpoint with stronger dense performance is used as a teacher to regularize later training stages, achieving strong empirical improvements. In contrast, we focus on understanding and mitigating the underlying optimization behavior itself, which we term semantic diffusion . Importantly, our approach is orthogonal to Gram anchoring and can potentially improve the quality of the teacher representations as well. CLS–Patch Interaction. The CLS token and patch tokens play inherently different roles but are typically processed through the same transformer blocks. Recent work suggests that ViTs implicitly attempt to separate these token types and that explicit CLS/patch specialization can improve dense representations ( 14 ) . Our findings are consistent with this perspective: modifying CLS–patch interaction appears more effective than directly constraining final patch representations. Unlike prior specialization approaches, we intervene at the attention normalization level rather than introducing separate processing branches or additional parameters. Sparse Attention. Sparse attention has primarily been explored for efficient sequence modeling and scalable transformers. Entmax and adaptive sparse transformers demonstrate that replacing softmax with sparse probability mappings can produce selective attention distributions while remaining fully differentiable ( 17 ; 5 ) . In ViTs, sparsity has often been introduced for computational or architectural efficiency ( 12 ; 3 ) . In contrast, we use sparsity as a representational bias rather than a compute optimization. Our goal is to make semantic diffusion less favorable during optimization by encouraging more selective token interactions. (a) Layer/epoch heatmap of point-in-box (PiB). (b) VOC dense-probe mIoU across pretraining epochs and feature layers. Figure 2: Relationship between CLS spatial grounding and dense prediction quality across training. Left: PiB heatmap showing how well CLS-aligned patches remain localized within foreground regions. Right: VOC dense probing performance across layers and epochs. 3 Method 3.1 Semantic Diffusion as an Optimization Shortcut We hypothesize that under global training objectives, ViTs exploit unrestricted token interactions to distribute class-relevant semantics across patch tokens in a locally inconsistent manner. This optimization shortcut can improve global image-level recognition while gradually degrading patch-level semantic structure, especially for dense prediction tasks. To study this behavior, we analyze point-in-box (PiB), defined as the proportion of images where the patch with the highest cosine similarity to the [CLS] token lies inside the foreground bounding box ( 20 ) . Intuitively, higher PiB indicates stronger spatial alignment between [CLS] semantics and foreground regions. As shown in Figure 2(a) , shallow layers consistently