PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

Tianchen Zhao, Ke Hong, Xinhao Yang, Xuefeng Xiao, Huixia Li, Feng Ling, Ruiqi Xie, Siqi Chen, Hongyu Zhu, Zhang Yichong, Yu Wang · 2025

Read PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models by Tianchen Zhao, Ke Hong, Xinhao Yang, Xuefeng Xiao, Huixia Li, Feng Ling, Ruiqi Xie, Siqi Chen, Hongyu Zhu, Zhang Yichong, Yu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16054.txt.

Introduction

Zhao and co-authors speed up image and video generation by reshuffling how the model's attention is laid out before trimming it. Attention cost grows steeply with sequence length, hurting high-resolution and multi-frame work. Trimming and compression tricks falter because visual attention patterns are scattered and irregular.

Their 2025 method instead reorders tokens so scattered patterns become tidy blocks that hardware likes. Both trimming and low-precision math then work cleanly on the unified pattern. The full system matched full-precision quality on image and video tests while using only a fifth to a third of the attention links and low bit widths. End-to-end speed rose about two to nearly three fold. The authors target efficient visual generation.

Catalog record

CREATORTianchen Zhao, Ke Hong, Xinhao Yang, Xuefeng Xiao, Huixia Li, Feng Ling, Ruiqi Xie, Siqi Chen, Hongyu Zhu, Zhang Yichong, Yu Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-16054

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models Tianchen Zhao, Ke Hong, Xinhao Yang, Xuefeng Xiao, Huixia Li, Feng Ling, Ruiqi Xie, Siqi Chen, Hongyu Zhu, Zhang Yichong, Yu Wang Abstract In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolution image or multi-frame video generation. To address this, prior research has explored techniques such as sparsification and quantization. However, these techniques face significant challenges under low density and reduced bitwidths. Through systematic analysis, we identify that the core difficulty stems from the dispersed and irregular characteristics of visual attention patterns. Therefore, instead of introducing specialized sparsification and quantization design to accommodate such patterns, we propose an alternative strategy: “ reorganizing ” the attention pattern to alleviate the challenges. Inspired by the local aggregatin nature of visual feature extraction, we design a novel Pattern-Aware token ReOrdering (PARO) technique, which unifies the diverse attention patterns into a hardware-friendly block-wise pattern. This unification substantially simplifies and enhances both sparsification and quantization. We evaluate the performance-efficiency trade-offs of various design choices and finalize a methodology tailored for the unified pattern. Our approach, PAROAttention , achieves video and image generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density ( 20%-30% ) and bitwidth ( INT8/INT4 ), achieving a 1.9 ∼ \sim 2.7 × \times end-to-end latency speedup. Figure 1 : PAROAttention unifies the diverse attention patterns through token reorder, which benefits both the sparsification and quantization. It achieves nearly identical generation result from full-precision baseline without metrics degradation, under lower density ( 20%-30% ) and bitwidth ( INT8/INT4 ), achieving a 1.9 \sim 2.7 \times end-to-end latency speedup. 1 1 footnotetext: All videos in the figure are provided in the supplementary. Figure 2 : The Motivation of PAROAttention. (a) The computational flow of transformer. (b) The challenge for sparsification and quantization caused by visual attention pattern, and how PAROAttention addresses it. (c) The illustration of 3D feature, and 1D token sequence with different orders. 1 Introduction Diffusion transformers (DiTs dit ) have garnered significant research interest in visual generation tasks. However, their excessive resource cost poses challenges for broader applications. The adoption of "3D full attention" in models like CogVideoX cogvideox further increases token length. The quadratic complexity of attention mechanisms in token length results in substantial memory consumption and computational overhead when processing such long sequences. For instance, generating a 49-frame 6-second 720P video involves 17K token length 2 2 2 We use this setting as example for most of the description below, the image token length N=17550=F*W*H=13*30*45 , where F,H,W stands for the frame number, height, and width in the latent space. . Attention computation contributes to the majority of the total latency, making it the primary bottleneck and requiring urgent optimization. As shown in fig. 2 (a), the two matrix multiplications QK^{T} and PV ( P is the attention map after softmax) has a computation cost quadratic to token length, making them the primary bottleneck in attention. Previous research has explored sparse attention mechanisms bigbird ; seerattention and quantization sageattention ; sageattention2 ; zhang2025sageattention3 to accelerate attention. While these techniques achieve notable success in language models MoBA ; NSA ; flex_prefill ; minference , they cannot be directly applied to visual generative models due to distinct attention patterns . As shown in fig. 2 -(b), different attention heads exhibit diverse patterns . These patterns vary not only in type (e.g., blockwise, multi-diagonal) but also in their structural characteristics—such as the number, width, and spacing of the diagonals. Recent sparse attention methods ditfastattn ; sta ; sparse_video_gen explore designing specialized sparse masks for visual models, but still face significant challenges in maintaining quality at lower density rates (< 50%). Quantization techniques tailored to visual generation remain underexplored. Existing methods struggle to efficiently quantize PV computations to lower-bit integers (e.g., INT8/INT4), and often remain at FP16/FP8. We conduct a systematic analysis of the underlying reason for subpotimal performance of existing techniques under low density and bitwidths (as discussed in section 3 ). Our findings reveal that the core challenge of sparsification and quantization stem from distinct characteristics of visual attention patterns. For sparsification, the dispersed and dynamically changing attention patterns lead to the absence of locally sparse regions, resulting in sparsification error. For quantization, the presence of multiple "diagonal" values act as outliers within data group, thereby increasing quantization error. Different from existing methods that design specialized sparsification and quantization techniques to accommodate the diverse patterns, we propose an alternative direction: To “reorganize” the attention patterns to ease the difficulty the design of both sparsification and quantization methodologies. To design proper technique to “reorganize” the attention pattern, we further analyze the underlying causes of the diverse visual attention patterns. Prior literature swin reveals the local aggregation nature of visual attention, which suggest that attention tends to capture relationships between neighboring pixels. In vision transformers, the 3D physical space are flattened into 1D token sequences, which disrupts the data adjacency. For example, as shown in fig. 2 -(c), the neighboring tokens along F -dimension in 3D space are not consecutive, and have the same interval of H \times W in 1D token sequence of default order [F,H,W] . The aggregation of these tokens with equal intervals forms the “multi-diagonal” pattern. Therefore, the multi-diagonal and block-wise patterns are intrinsically the same, representing local aggregation in different dimensions. By applying token reordering, we rearrange the tensor layout (e.g., permute from [F,H,W] to [H,W,F] )) for each head to keep elements of the local aggregation dimension contiguous. The “Pattern-Aware token ReOrder (PARO)” could transform diverse patterns into a unified, hardware-friendly block-wise pattern. Furthermore, we analyze the trade-offs among design choices in terms of accuracy, and efficiency, and design specialized sparsification and quantization techniques tailored to the unified block-wise pattern, constructing the PAROAttention method. We summarize our contributions as follows: • We analyze and identify the key challenges of attention sparsification and quantization as unique attention pattern characteristic, and propose to address it from a novel direction of token reorder to reorganize and unify the attention pattern. • We compare the strength and weakness of design choices, and develop specialized methodologies tailored for the unified pattern, along with CUDA kernels for practical acceleration. • PAROAttention, achieves generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density ( 20%-30% ) and bitwidth ( INT8/INT4 ), achieving a 1.9 \sim 2.7 \times end-to-end latency speedup. 2 Related Works 2.1 Visual Generative Models Diffusion transformers dit , which leverage transformer architectures transformer ; vit , have achieved outstanding performance and are widely adopted by recent image generation models such as PixArt pixart-alpha and Flux flux . For video generation, earlier approaches, such as OpenSORA open-sora , apply "spatial-temporal" attention, which performs attention separately along the spatial and temporal dimensions. Other recent models like CogVideoX cogvideox , Wan wan2.1 adopt “3D full attention" instead, processing all spatial tokens across all frames simultaneously. The increased model size and token length pose significant challenges for efficient deployment. 2.2 Sparse Attention for Generative Models Existing sparse attention research MoBA

Read the complete text