Taming Outlier Tokens in Diffusion Transformers

Xiaoyu Wu Yifei Wang Tsu-Jui Fu Liang-Chieh Chen Zhe Gan Chen Wei, [0.6em] Rice University · 2026

Read Taming Outlier Tokens in Diffusion Transformers by Xiaoyu Wu Yifei Wang Tsu-Jui Fu Liang-Chieh Chen Zhe Gan Chen Wei, [0.6em] Rice University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05206.txt.

Introduction

Xiaoyu Wu and five co-authors study outlier tokens in diffusion transformers used for image generation, a few tokens with unusually large values that attract heavy attention while carrying little local information. The phenomenon was known in vision transformers but not in generative models. The authors find it in both the encoder and the denoising transformer of modern pipelines.

Simply masking the large tokens does not help, suggesting the real problem is corrupted local patch meaning rather than extreme values alone. Their Dual-Stage Registers add register tokens at both stages, trained where possible and created at test time otherwise. Across image benchmarks the intervention reduced outlier artefacts and improved generation quality.

Catalog record

CREATORXiaoyu Wu Yifei Wang Tsu-Jui Fu Liang-Chieh Chen Zhe Gan Chen Wei, [0.6em] Rice University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05206

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Taming Outlier Tokens in Diffusion Transformers Xiaoyu Wu Yifei Wang Tsu-Jui Fu Liang-Chieh Chen Zhe Gan Chen Wei, [0.6em] Rice University Abstract We study outlier tokens in Diffusion Transformers (DiTs) for image generation. Prior work has shown that Vision Transformers (ViTs) can produce a small number of high-norm tokens that attract disproportionate attention while carrying limited local information, but their role in generative models remains underexplored. We show that this phenomenon appears in both the encoder and denoiser of modern Representation Autoencoder (RAE)-DiT pipelines: pretrained ViT encoders can produce outlier representations, and DiTs themselves can develop internal outlier tokens, especially in intermediate layers. Moreover, simply masking high-norm tokens does not improve performance, indicating that the problem is not only caused by a few extreme values, but is more closely related to corrupted local patch semantics. To address this issue, we introduce Dual-Stage Registers (DSR) , a register-based intervention for both components: trained registers when available, recursive test-time registers otherwise, and diffusion registers for the denoiser. Across ImageNet and large-scale text-to-image generation, these interventions consistently reduce outlier artifacts and improve generation quality. Our results highlight outlier-token control as an important ingredient in building stronger DiTs. * * footnotetext: Equal Contribution 1 Introduction Vision Transformers (ViTs) [ 7 ] have emerged as a foundational architecture in computer vision. Initially, they demonstrated strong performance on visual understanding tasks, including supervised classification (e.g., ViT [ 7 ] , DeiT [ 32 ] ), self-supervised representation learning (e.g., DINO models [ 3 , 18 , 25 ] ), and contrastive language-image pretraining (e.g., CLIP [ 22 ] , SigLIP [ 38 , 34 ] ). These successes gradually shifted the default backbone for visual recognition models from convolutional networks such as ResNets [ 10 ] to Vision Transformers [ 4 , 23 ] . More recently, ViTs have shown remarkable effectiveness in generative modeling, particularly for diffusion models [ 27 , 11 , 28 ] . Starting from DiT [ 20 ] , Transformer-based generators have increasingly replaced UNet architectures [ 11 , 29 , 24 ] in both latent-space [ 17 , 8 ] and pixel-space diffusion models [ 16 ] . Beyond the generator itself, recent work such as RAE [ 39 ] further proposes using pretrained ViT encoders to project images into latents, replacing the UNet-based VAE [ 24 ] , the last remaining convolutional component in state-of-the-art diffusion pipelines. A commonly cited advantage of Transformers is their ability to model global interactions and their favorable optimization properties, albeit at the cost of weaker inductive biases. Recent studies, however, show that this flexibility can introduce nontrivial artifacts in the feature maps. In particular, Vision Transformers Need Registers [ 5 ] identifies the ubiquitous presence of outlier tokens in supervised and self-supervised ViTs (e.g., DeiT-III [ 33 ] , OpenCLIP [ 13 ] , DINOv2 [ 18 ] ). These tokens exhibit abnormally large norms, absorb a disproportionate amount of attention, and yet carry little semantic information, resulting in artifacts in feature maps that hinder downstream utilization, especially for dense visual tasks. This behavior is closely related to the attention sink hypothesis in the context of large language models [ 35 ] , and can be effectively mitigated by introducing register tokens, i.e., dedicated non-patch tokens that explicitly serve as attention sinks during training. However, investigations of outlier tokens have thus far focused almost exclusively on recognition models, leaving their role in generative models largely unexplored. This gap is particularly striking given the growing prevalence of Transformers in generative modeling and the heightened importance of locality and spatial fidelity in generation compared to recognition. In this work, we show that outlier token effects are pervasive in both ViT-based autoencoder and diffusion Transformers. For ViT-based autoencoders, we focus on RAE [ 39 ] , which has pretrained representation encoders. We empirically confirm the existence of outlier tokens in the encoders and demonstrate their negative impact on generation quality. Using the official DINOv2 implementation [ 18 , 5 ] , we compare DINOv2 encoders with and without register tokens and observe consistent improvements when registers are enabled. For SigLIP 2 [ 34 ] , which does not include registers by default, we introduce recursive test-time registers [ 14 ] and show that this modification, too, leads to improved generation. For Transformer-based generators, identifying outlier tokens is more subtle. In supervised and self-supervised ViTs, inputs and outputs differ in semantic abstraction, and prior work reports that outlier tokens typically emerge in late layers. In contrast, diffusion models operate on inputs and outputs of the same modality—local pixels or latents. Surprisingly, we find that outlier tokens still arise, but predominantly in intermediate layers of the diffusion Transformer. By introducing register tokens into the generator, we achieve consistent gains across a variety of diffusion architectures, including SiT [ 17 ] , JiT [ 16 ] , and RAE-based [ 39 , 31 ] designs. We refer to this unified use of register tokens in both the representation encoder and the diffusion generator as Dual-Stage Registers (DSR). Overall, our results show that DSR provide a simple yet effective way to stabilize Transformer-based diffusion pipelines by addressing outlier tokens in both the tokenizer and the denoiser. This unified intervention consistently improves generation quality across settings. Concretely, for RAE-DiT with SigLIP2-B, it reduces ImageNet-256 FID from 5.89 to 4.58 and improves GenEval from 0.426 to 0.466 on a large-scale text-to-image task. 2 Related Work Outlier Tokens in Transformers. In Vision Transformers, Darcet et al. identify a small fraction of patch tokens with unusually large feature norms, referred to as outlier tokens [ 5 ] . These tokens often carry limited local patch information while inducing irregular attention patterns. To mitigate this effect, they introduce extra learnable register tokens during training, which absorb the outlier behavior and lead to cleaner attention over image patches [ 5 ] . Building on this finding, subsequent work traces these outliers to a sparse set of register neurons and proposes a training-free alternative: at inference time, the corresponding activations can be shifted into an additional untrained token, recovering much of the benefit of trained registers in models released without them [ 14 ] . Related sink-like phenomena have also been studied in language models. In this setting, attention sinks attract disproportionate attention and have been linked to large residual-stream activations, motivating neuron-level or activation-level test-time interventions [ 35 , 30 , 36 ] . More recent work explores the connection between sink behavior and outlier-driven dynamics [ 21 ] . Together, these studies suggest that outlier or sink tokens are a recurring phenomenon in Transformer architectures, and register-augmented ViT backbones have become increasingly common in practice [ 5 ] . By contrast, in Diffusion Transformers (DiTs), it remains unclear whether similar outlier tokens emerge during denoising and whether they affect generation quality. Our work addresses this gap. Diffusion Transformers Across Different Input Spaces. DiTs are originally trained in variational autoencoder (VAE) latent spaces for efficiency, but the tokenizer can bottleneck reconstructed image quality because its latents may not fully preserve semantic information [ 24 , 20 ] . Subsequent work improves latent DiT pipelines through representation alignment with visual features from vision foundation models, improving training

Read the complete text