Attention Sinks in Diffusion Transformers: A Causal Analysis

Fangzheng Wu, Brian Summa · 2026

Read Attention Sinks in Diffusion Transformers: A Causal Analysis by Fangzheng Wu, Brian Summa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09313.txt.

Description

Wu and a colleague present a causal analysis of attention sinks in text-to-image diffusion transformers, testing whether tokens with a lopsided share of attention matter for the model. In autoregressive language models, sinks are assumed to matter a lot, but their role in diffusion transformers was unclear. The team spots the top attention getters at each step and quiets them with paired actions on score and value paths, needing no training.

On 553 GenEval prompts on Stable Diffusion 3, checked on SDXL, removing sinks does not hurt text-image match or preference scores at k equals 1. Only under stronger actions at k at least 10 does HPS-v2 show a metric dependent boundary; CLIP-T stays robust. Shifts in what viewers see when sinks are shut off vary by sink, about 6 times larger than equal budget random masking. This shows a split between trajectory level perturbation and semantic alignment.

Catalog record

CREATORFangzheng Wu, Brian Summa
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09313

Browse the full library · About the Tower of Babel