VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation

Yupeng Sun · 2026

Read VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation by Yupeng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12798.txt.

Introduction

Yupeng Sun and seven colleagues speed up FlashAttention by easing vector work. Online softmax streams score tiles and tracks a running maximum. Near peak throughput, rowmax and rowsum reductions dominate latency. Vector Relieved Flash Attention seeds the maximum from key-block summaries. It visits sink and local blocks first, then freezes the maximum. A sparse variant adds block-sparse skipping. Both avoid the conditional rescale step of FlashAttention 4.0. Tests show no performance loss. One setting nearly doubles speed over the baseline.

Catalog record

CREATORYupeng Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63404
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12798

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation Yupeng Sun Abstract Abstract FlashAttention-style online softmax enables exact attention computation with linear memory by streaming score tiles through on-chip memory and maintaining a running maximum/normalizer [ 6 , 8 ] . However, as attention kernels approach peak tensor-core/cube-core throughput on modern accelerators, the non-matmul components of online softmax—in particular, per-tile rowmax / rowsum reductions and the rescale chain—can become vector/SIMD limited and dominate end-to-end latency. This paper revisits FlashAttention and proposes Vector Relieved Flash Attention (VFA), a hardware-friendly modification that reduces the frequency of rowmax -driven updates of the running maximum while preserving the online-softmax accumulation structure. VFA initializes the running maximum using a cheap approximation based on key-block representations, reorders the key-block traversal to prioritize empirically high-impact regions (sink and local blocks), and freezes the running maximum for the remaining blocks to avoid repeated reductions and rescale operations. We further show that VFA composes naturally with online block-sparse skipping methods such as BLASST [ 26 ] , yielding Vector Relieved Sparse Attention (VSA) that combines fewer processed blocks with cheaper per-block load overhead. It’s noteworthy that VFA/VSA entirely bypasses the rescale operation in the update stage, an operation which is conditionally executed in FA4.0. Extensive evaluations on representative benchmarks (e.g., MMLU, MATH500) and numerical analyses of attention statistics validate the design choices: (i) sink+local reordering is supported by early stabilization of the running maximum, (ii) naive block summaries of Q/K are insufficient due to intra-block heterogeneity, and (iii) m -initialization is necessary to cover cases where maxima occur in middle blocks. Overall, VFA/VSA provide a practical path to improving attention efficiency in regimes where online-softmax reductions become a primary kernel bottleneck, without degrading the model performance. Compared to the baseline C16V32, the C8V32, C4V32 and C4V16 versions can achieve about 2x speedup based on the latest architecture and they have all reached the vector bottleneck. As the evolution of new architecture, C4V16 will deliver a 6x speedup by increasing the exponent capacity. 1 Introduction Transformer attention is a core primitive for modern foundation models across language, vision, and multimodal settings [ 25 , 19 , 9 ] . Yet, scaled dot-product attention is notoriously expensive for long sequences due to its quadratic complexity and the associated memory traffic for softmax and PV [ 25 ] . This has motivated substantial work on efficient attention mechanisms, encompassing two main directions: efficient kernel design and low-complexity algorithm design. Low complexity algorithm design primarily includes linear approximations and sparse computation patterns [ 23 ] . FlashAttention is a typical representative of efficient kernel design methods and has emerged as a widely adopted solution for exact attention: it avoids materializing the full attention matrix in HBM by tiling Q/K/V into on-chip memory and updating softmax statistics online, achieving linear memory in sequence length while retaining numerical stability [ 6 ] . The FlashAttention line of work further demonstrates that efficiency is not purely an algorithmic issue but also a kernel-mapping problem. FlashAttention-2 improves performance via better parallelism and work partitioning, increasing occupancy and reducing shared-memory communication [ 8 ] . FlashAttention-3 exploits hardware features such as asynchronous data movement and low-precision pathways (e.g., FP8) to further improve pipeline efficiency on modern GPUs [ 21 ] . FlashAttention-4 further reduces softmax-side overhead by approximating exp2 with low-latency linear instructions and conditionally eliding the output rescale on \tilde{O} when it is numerically safe, thereby shrinking the non-matmul critical path [ 7 ] . These advances underline a key trend: once tiled matmuls are highly optimized, attention performance increasingly hinges on the remaining non-matmul components of online softmax, including per-tile reductions ( rowmax / rowsum ) and the rescale chain used to maintain numerical stability. Motivation: when online-softmax becomes vector limited. To achieve higher throughput and lower latency, models are evolving toward lower precision such as MXFP4 [ 24 ] , NVFP4 [ 1 ] and HIF4 [ 16 ] . However, these lower precisions are only applied in GEMM and vector operations are still in high precision, which bring higher load imbalance for fused kernels. In many practical deployments, we observe that the cost of online-softmax statistic updates can become vector/SIMD limited and disproportionately impacts end-to-end latency, especially when matmul pipelines already run near peak utilization. This motivates a complementary research question to prior FlashAttention work: can we reduce the frequency of per-tile reductions and rescale operations, while keeping the same online-softmax accumulation structure and kernel compatibility? Our approach. We propose VFA, a hardware-friendly variant of FlashAttention-style online softmax. VFA modifies the standard per-tile update schedule in three steps. First, it performs a fast m -initialization that approximates the per-row running maximum using cheap dot products between Q_{i} and precomputed key-block representations. Second, it reorders the traversal of key blocks to prioritize an initial “sink” block and the local block aligned with the query position, motivated by empirical evidence that attention mass and the eventual running-max stabilization concentrate in these regions. Third, it freezes the running maximum for the remaining blocks, skipping per-block rowmax and the rescale chain, while still accumulating \exp(S_{ij}-m_{i}) and PV using the same state variables and stabilization form. Importantly, VFA targets statistic-computation overhead rather than skipping attention blocks outright. Composition with FlashAttention-4. VFA naturally skip all the rescale operations in attention computation, avoiding online conditional branch in FA4 [ 27 ] . Composition with sparse attention. VFA is orthogonal to sparse attention methods that reduce the number of processed blocks. In particular, BLASST introduces an online-statistics-driven rule that skips blocks whose local maximum is sufficiently below the running maximum, requiring minimal changes to FlashAttention-style kernels [ 26 ] . We show that VFA composes naturally with BLASST, yielding VSA that combines two multiplicative gains: fewer blocks processed (sparsification) and cheaper per-block load overhead (vector-relieved online softmax). Empirical analyses supporting the design. Beyond end-to-end accuracy evaluations, we provide numerical analyses of attention statistics to justify each component. We show that the running maximum often stabilizes early at sink/local blocks, explaining why reordering is effective. We also demonstrate that naive block representations of Q/K are ineffective due to moderate intra-block directional similarity and large norm variability, and that m -initialization is necessary because the true block-maximum can occur in middle blocks that would otherwise be missed under max-freezing. Contributions. In summary, this work makes the following contributions: • We identify and study a kernel-level bottleneck in FlashAttention-style online softmax: per-tile reductions and rescale operations can become vector/SIMD limited in regimes where matmul is already efficient. • We propose VFA, a hardware-friendly online-softmax variant that uses m -initialization, sink+local reordering, and max-freezing to reduce the frequency of rowmax -driven updates while preserving the standard accumulation structure. • We present VSA by composing VFA with BLASST-style block skipping [ 26 ] , illustrating that vector-relieved load overhead and sparse skipping are complementary, and suggesting that VFA can serve as a reusable building block in future attention kernel evolution. • We provide numerical analyses of attention statistics (running-max stabilization, block similarity, norm variability, and block-max peak locations) that explain when and why each design choice is effective. • We evaluate the performance of VFA based on the compute capability of the latest architecture. 2

Read the complete text