QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm
Qirui Zhou, Shaohui Peng, Weiqiang Xiong, Haixin Chen, Yuanbo Wen, Haochen Li, Ling Li, Qi Guo, Yongwei Zhao, Ke Gao, Ruizhi Chen, Yanjun Wu, Chen Zhao, Yunji Chen · 2025
Read QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm by Qirui Zhou, Shaohui Peng, Weiqiang Xiong, Haixin Chen, Yuanbo Wen, Haochen Li, Ling Li, Qi Guo, Yongwei Zhao, Ke Gao, Ruizhi Chen, Yanjun Wu, Chen Zhao, Yunji Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12355.txt.
Introduction
Qirui Zhou and thirteen co-authors report a method for letting AI write its own fastest code in a 2025 paper. Attention is the core operation inside large language models, and speeding it up for long inputs is a known bottleneck. The best hand-written version must be reworked for each graphics chip, which is slow and costly.
Models are good at writing ordinary code but poor at this low-level craft, since they struggle with data flow and chip tricks. The team gives the model an in-between language that separates the plan of the computation from chip details. A two-step process then turns plans into code and code into chip-specific form. On three different graphics chips, the generated code far beat what plain models produced. The paper frames this as a self-improving loop for the very operation the models themselves run on.
Catalog record
| CREATOR | Qirui Zhou, Shaohui Peng, Weiqiang Xiong, Haixin Chen, Yuanbo Wen, Haochen Li, Ling Li, Qi Guo, Yongwei Zhao, Ke Gao, Ruizhi Chen, Yanjun Wu, Chen Zhao, Yunji Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-12355
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
QiMeng-Attention: SOTA Attention Operator is generated by SOTA Attention Algorithm Qirui Zhou, Shaohui Peng, Weiqiang Xiong, Haixin Chen, Yuanbo Wen, Haochen Li, Ling Li, Qi Guo, Yongwei Zhao, Ke Gao, Ruizhi Chen, Yanjun Wu, Chen Zhao, Yunji Chen Abstract The attention operator remains a critical performance bottleneck in large language models (LLMs), particularly for long-context scenarios. While FlashAttention is the most widely used and effective GPU-aware acceleration algorithm, it must require time-consuming and hardware-specific manual implementation, limiting adaptability across GPU architectures. Existing LLMs have shown a lot of promise in code generation tasks, but struggle to generate high-performance attention code. The key challenge is it cannot comprehend the complex data flow and computation process of the attention operator and utilize low-level primitive to exploit GPU performance. To address the above challenge, we propose an LLM-friendly Thinking Language (LLM-TL) to help LLMs decouple the generation of high-level optimization logic and low-level implementation on GPU, and enhance LLMs’ understanding of attention operator. Along with a 2-stage reasoning workflow, TL-Code generation and translation, the LLMs can automatically generate FlashAttention implementation on diverse GPUs, establishing a self-optimizing paradigm for generating high-performance attention operators in attention-centric algorithms. Verified on A100, RTX8000, and T4 GPUs, the performance of our methods significantly outshines that of vanilla LLMs, achieving a speed-up of up to 35.16 × 35.16\times . Besides, our method not only surpasses human-optimized libraries (cuDNN and official library) in most scenarios but also extends support to unsupported hardware and data types, reducing development time from months to minutes compared with human experts. 1 Introduction The attention mechanism is the cornerstone of modern Large Language Models (LLMs). Its time and memory complexity grows quadratically in sequence length, and attention has emerged as the bottleneck for the runtime and memory resource requirements for LLMs, particularly in long-context scenarios Dao et al. 2022 . Consequently, numerous acceleration algorithms have been proposed to accelerate attention operators, with FlashAttention being the most effective and widely used Dao et al. 2022 ; Dao 2024 ; Hong et al. 2024 . With the rise of cost-efficient, high-performance LLMs like DeepSeek DeepSeek-AI et al. 2024a ; DeepSeek-AI et al. 2024b ; DeepSeek-AI et al. 2025 , deploying LLMs across diverse GPU specifications (including legacy generations) has become increasingly prevalent. Figure 1: Our LLM-TL enables LLMs to generate high-performance Tensor Core kernel for attention. However, due to the complexity of attention operators and disparities in the architectures and instruction sets across different GPUs, the efficient implementation and migration of FlashAttention algorithms becomes a pressing challenge. Existing methods for implementing high-performance attention operators can be divided into two categories: i.e., manual implementation and deep learning compilers. Human experts optimize attention operators through fused computation based on their expertise in GPU architecture and algorithm knowledge, thereby effectively enhancing computation efficiency. Consequently, these hand-crafted implementations are time-consuming and non-portable, failing to support some common GPUs and datatypes (e.g., RTX8000 GPU and FP8 datatype). In addition, existing mainstream deep learning compilers (e.g., TVM Chen et al. 2018 , Ansor Zheng et al. 2020 ) fail to automatically implement FlashAttention due to their inability to perform optimizations involving fused sequences of multiple complex operations (e.g. GEMM). Overall, current methods struggle to automatically analyze the attention operator characteristics and GPU architecture to efficiently generate high-performance FlashAttention implementation on different GPU platforms, which limits the adaptability of LLMs. Motivated by strong code-generation abilities of LLMs Bairi et al. 2024 ; Zhong et al. 2024 ; Holt et al. 2024 ; Li et al. 2024 ; Wang et al. 2023 ; Zhou_Wen_Chen_Gao_Xiong_Li_Guo_Wu_Chen_2025 ; zhang2025qimengtensoropautomaticallygeneratinghighperformance , it is a promising direction to tackle these issues by utilizing LLMs automatically implement FlashAttention on GPUs. Although LLMs can create attention implementations by high-level APIs like PyTorch Paszke and et al. 2019 , they have trouble generating high-performance FlashAttention with low-level primitives (e.g., CUDA or CuTe NVIDIA/CUTLASS 2023 ) with crucial optimization techniques like data blocking and fused computation. This significantly hinders the reduction of memory access overheads and the enhancement of computational efficiency, as shown in Figure 1 (a).The primary challenges preventing LLMs from generating high-performance FlashAttention implementations in one pass are twofold: (1) LLMs cannot comprehend the complex data flow and multi-step computation of attention, thus failing to generate effective optimization logic; (2) LLMs are unable to produce optimized implementations based on low-level primitives to fully exploit GPU characteristics. Inspired by human experts’ decoupling of optimization and implementation, we propose an LLM-friendly Thinking Language (LLM-TL) to address the above challenge by helping LLMs decouple the generation of abstract optimization logic and low-level code implementation on GPU. Specifically, LLM-TL encompasses two types of statements and their requisite parameters: memory access and computation, enabling LLMs to comprehend the data flow and computation processes of the attention operator on GPU architecture from the vantage point of high-level abstract semantics, as shown in Figure 1 (c). Based on LLM-TL, we further propose an automated workflow, leveraging the reasoning capabilities of LLMs to implement the high-performance FlashAttention operator on GPUs with low-level primitives.Formally, the workflow includes: 1) TL Code generation: LLMs use LLM-TL statements to create sketch code to represent the execution flow on GPU, then reasoning and filling the parameters needed for each statement; 2) TL Code translation: LLMs convert the TL-code into low-level CUDA code based on the target GPU architecture and instruction set. Through LLM-TL and the workflow, LLMs can efficiently and automatically implement the high-performance FlashAttention on different GPUs. The contributions of this paper are as follows: (1) We propose LLM-TL, an LLM-friendly abstract language, enabling LLMs to deeply comprehend and optimize complex attention operators on GPUs, thereby establishing a self-optimizing paradigm for generating high-performance attention operators in attention-centric algorithms. (2) We present an automated workflow that enables LLMs to leverage LLM-TL to implement high-performance FlashAttention on diverse GPUs, which extends the capabilities of LLMs. (3) Verified on A100, RTX8000, and T4 GPUs, the performance of FlashAttention implementation generated by our methods significantly outshines that of vanilla large language models (LLMs), achieving a speed-up of up to 35.16\times , and also outperforms with cuDNN or official implementation in most cases. Moreover, the development time also be reduced from months to minutes compared with human experts. 2 Preliminary 2.1 GPU Characteristics The rapid development of AI is closely tied to the advancement of GPU’s high parallel computing power. GPUs achieve high parallelism by stacking a large number of simple computation units. They use the SIMT (Single Instruction, Multiple Threads) design, which allows for the execution of many threads running the same code. The introduction to the GPU architecture is illustrated in Figure 2 . 2.1.1 Memory Hierarchy The memory hierarchy of GPUs is divided into global memory, shared memory, and registers. Global memory is shared by all threads, shared memory is shared by threads within the same thread block, and each thread has its own private registers. Due to this hierarchical memory structure, explicit management of data movement between different memory levels is required during programming. Data stored in registers can be directly accessed for fast computations, such as General Matrix Multiplication (GEMM). 2.1.2 Computation Unit To get better performance on GPU, we use Tensor Core Markidis et al. 2018 , which is a specialized processing unit