DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers

Hanling Zhang, Rundong Su, Zhihang Yuan, Pengtao Chen, Mingzhu Shen, Yibo Fan, Shengen Yan, Guohao Dai, Yu Wang, Tsinghua University · 2025

Read DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformers by Hanling Zhang, Rundong Su, Zhihang Yuan, Pengtao Chen, Mingzhu Shen, Yibo Fan, Shengen Yan, Guohao Dai, Yu Wang, Tsinghua University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22796.txt.

Description

Hanling Zhang and co-authors speed up the attention step inside multimodal diffusion transformers, the engines behind text-to-image tools. Their 2025 method answers a cost problem: attention math dominates run time and blocks scale. It works after training, so no retraining is needed.

The team found attention patterns that differ by head, unlike older image models. Their fix squeezes each head on its own terms and caches results for reuse. A custom fused software kernel adds further gains. Search for the best squeeze settings takes minutes, not hours, thanks to local measures. On two-thousand-pixel image generation, attention cost fell by sixty-eight percent and end-to-end speed rose one and a half times, with image quality held.

Catalog record

CREATORHanling Zhang, Rundong Su, Zhihang Yuan, Pengtao Chen, Mingzhu Shen, Yibo Fan, Shengen Yan, Guohao Dai, Yu Wang, Tsinghua University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22796

Browse the full library · About the Tower of Babel