FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference

Aniruddha Nrusimha, William Brandon, Mayank Mishra, Yikang Shen, Rameswar Panda, Jonathan Ragan-Kelley, Yoon Kim, [0.5em] Massachusetts Institute of Technology · 2025

Read FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference by Aniruddha Nrusimha, William Brandon, Mayank Mishra, Yikang Shen, Rameswar Panda, Jonathan Ragan-Kelley, Yoon Kim, [0.5em] Massachusetts Institute of Technology (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22758.txt.

Introduction

Aniruddha Nrusimha and co-authors present, in 2025, a way to run large language models faster when requests arrive one at a time. Most optimized software targets big batches, where computing power dominates. In low-batch settings, memory traffic and the overhead of launching many small operations dominate instead, and those cases matter for edge devices and snappy services.

Their system, FlashFormer, fuses the model's entire forward pass into a single program kernel, removing launch overheads and streamlining memory flow. Across model sizes and compression settings, it delivered solid speedups over existing inference kernels. The work argues whole-model fusion is the right shape for low-batch serving.

Catalog record

CREATORAniruddha Nrusimha, William Brandon, Mayank Mishra, Yikang Shen, Rameswar Panda, Jonathan Ragan-Kelley, Yoon Kim, [0.5em] Massachusetts Institute of Technology
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22758

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference Aniruddha Nrusimha, William Brandon, Mayank Mishra, Yikang Shen, Rameswar Panda, Jonathan Ragan-Kelley, Yoon Kim, [0.5em] Massachusetts Institute of Technology Abstract The size and compute characteristics of modern large language models have led to an increased interest in developing specialized kernels tailored for training and inference. Existing kernels primarily optimize for compute utilization, targeting the large-batch training and inference settings. However, low-batch inference, where memory bandwidth and kernel launch overheads contribute are significant factors, remains important for many applications of interest such as in edge deployment and latency-sensitive applications. This paper describes FlashFormer , a proof-of-concept kernel for accelerating single-batch inference for transformer-based large language models. Across various model sizes and quantizations settings, we observe nontrivial speedups compared to existing state-of-the-art inference kernels. † † Code: https://github.com/aninrusimha/flashformer 1 Introduction As the utility and popularity of deep learning models has increased dramatically over the past decade and a half, there has been an increasing interest in low-level kernel design , i.e. developing specialized programs tailored to optimize particular workloads on the target hardware. This has been particularly true for transformer and transformer-like architectures. This is due to a dramatic increase in task performance across domains for these models as well as the increased model scaling (i.e. increasing parameter counts and computational costs per forward pass) and inference scaling (i.e. generating more tokens) needed to achieve these results. While most accelerator vendors develop high performance kernels for common linear algebra routines such as matrix multiplication, these general-purpose kernels are often not sufficient for LLM inference. Consequently, there has been much recent work on kernels that accelerate the inference of transformer-based LLMs, such as FlashDecoding [ Dao et al. 2023 ] and PagedAttention [ Kwon et al. 2023 ] . These kernels achieve efficiency gains by optimizing parts of operations common to the transformer inference pipeline. For example, FlashDecoding modifies the work partitioning in FlashAttention [ Dao et al. 2022 ] for better parallelization during transformer decoding, while PagedAttention saves GPU memory by organizing the KV cache, thus allowing larger batch sizes. Figure 1 : Depiction of the key benefit of FlashFormer . The top half depicts normal Transformer execution with multiple kernels. After all chunks of Layer 1 have been loaded into the SM cache, we cannot start loading Layer 2 (top left). This, coupled with kernel overhead, leads to a break in memory loads and computation (top right). With FlashFormer , we fuse kernels to enable overlapping across prior kernel barriers (bottom left). This leads to more efficient overlapping and faster runtime (bottom right). Most kernels are optimized to maximize throughput for large matrix-matrix multiplications. The key to doing so on modern accelerators is to maximize the utilization of special matrix-multiply units (MMU)—called Tensor Cores on NVIDIA hardware and MXUs on TPUs—which provide most of the computational performance of the chip. In either large-batch inference or training, this design choice optimizes the primary bottleneck of these kernels, i.e. the computational cost of matrix multiplications with large arithmetic intensity (i.e., computation per element of data movement). However, many use cases operate under different constraints. For example, low-batch inference is primarily bottlenecked by the memory bandwidth costs of parameter and cache movement and depending on the size of the model, kernel launch overheads that arise from launching multiple separate kernels for each transformer layer. Low-batch inference workloads arise in many real-world applications, such as edge deployment, latency-sensitive applications, and more recently, inference-time scaling via longer rollouts. Moreover, low-batch inference does not obtain significant benefits from using MMUs on the hardware since the problem sizes are not bottlenecked by compute. Writing fast kernels for these workloads requires further innovation along two common directions in kernel development research: 1) Increased operator fusion, or packing an increasing number of instructions and operations into a single kernel and 2) Increased specialization, or generating kernels that more closely optimized for the particular problem. This paper describes a strategy for writing a highly optimized inference kernel that runs the entire forward pass of a transformer-based large language model in a single kernel, and is specialized for a particular model configuration (such as the number of attention heads, embedding dimension etc) and the target hardware architecture. We use meta-programming, a shared pipelined buffer, and fast synchronization mechanisms to create a fully fused and specialized kernel. The resulting kernel, which we call FlashFormer , exploits increased overlapping of memory movement and the single fused kernel amortizes the launch overhead for the kernel over the entire forward pass. We apply this kernel to accelerate single-batch inference of Llama models [ Grattafiori et al. 2024 ] , and find that it offers respectable speedups over state-of-the-art decoding kernels. 2 Background 2.1 Transformer Architecture We target the Llama-3 [ Grattafiori et al. 2024 ] family of models, which make use of the transformer architecture with pre-RMSNorm [ Jiang et al. 2023 ] , Grouped Query Attention [ Ainslie et al. 2023 ] , RoPE [ Su et al. 2021 ] , and SwiGLU FFN layers [ Shazeer 2020 ] . Each layer of a transformer model consists of six matrix operations, commonly grouped into five sublayers. Each sublayer has one matrix operation, except for the ‘‘core attention’’ layer, 1 1 1 Since “attention layer” could be ambiguous, we use “core attention” to refer to the operation \operatorname{softmax}(\alpha{\bm{Q}}{\bm{K}}^{\top}){\bm{V}} . which has two. The matrix operations per layer with their respective input and output activations are listed in Table 1 . During autoregressive token-by-token generation, each matrix operation takes as input a large matrix stored in global memory as well as an activation computed by a previous operation. Transformer models are parameterized by their number of layers, model dimension, FFN dimension, head dimension, and the number of attention and key-value heads. The models we generate kernels for in this work are shown in Table 2 . Matrix Operation Matrix Load Input Output \bm{QKV} Projection {\bm{W}}_{\bm{QKV}} \text{RMSNorm}({\bm{x}}) {\bm{Q}}[i],{\bm{K}}[i],{\bm{V}}[i] {\bm{Q}}{\bm{K}}^{\top} computation {\bm{K}}[\hskip 0.85358pt:\hskip-1.99168pti] {\bm{Q}}[i] {\bm{S}}[i] \bm{PV} computation {\bm{V}}[\hskip 0.85358pt:\hskip-1.99168pti] \text{softmax}(\alpha{\bm{S}}[i]) {\bm{O}} Attention Output Projection {\bm{W}}_{{\bm{a}}_{out}} \text{vector}({\bm{O}}) {\bm{a}}_{out} GLU In/Gate Projection {\bm{W}}_{FFN1} \text{RMSNorm}({\bm{x}}+{\bm{a}}_{out}) {\bm{g}}_{inter} GLU Output Projection {\bm{W}}_{FFN2} \text{SwiGLU}({\bm{g}}_{inter}) {\bm{g}}_{out} Table 1 : Llama matrix operations for decoding. Here i is the current sequence position, and {\bm{x}} is the residual stream. {\bm{W}}_{\bm{QKV}},{\bm{W}}_{{\bm{a}}_{out}}{\bm{W}}_{FFN1} and {\bm{W}}_{FFN2} represent corresponding parameter matrices, and {\bm{Q}},{\bm{K}} and {\bm{V}} represent the query, key, and value matrices respectively. {\bm{S}} represents the attention logits, {\bm{P}} the attention weights, and {\bm{a}}_{out} the output of attention. {\bm{g}}_{inter} and {\bm{g}}_{out} represent the output of the GLU input/gate projection and the final GLU output of the GLU layer respectively. Model Layers d_{model} d_{inter} d_{head} n_{q\_heads} n_{kv\_heads} Llama 3.1 8B 32 4096 14336 128 32 8 Llama 3.1 70B 80 8192 28672 128 64 8 Table 2 : Configuration of Llama 3.1 8B and 70B, for which we develop FlashFormer kernel. 2.2 GPU Programming and Architecture GPU programming optimizes over three interleaved hierarchies: the program hierarchy, the hardware hierarchy, and the memory hierarchy. The program hierarchy structures computations into kernels, thread blocks, warps, and threads to effectively expose parallelism. 2 2 2 Note that we use the NVIDIA GPU hardware terminology in this section, other architectures use different terms for similar hardware components. The hardware hierarchy maps

Read the complete text