SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimization

Arya Tschand, Muhammad Awad, Ryan Swann, Kesavan Ramakrishnan, Jeffrey Ma, Keith Lowery, Ganesh Dasika, Vijay Janapa Reddi, [6pt] Harvard University, [6pt] [email protected] · 2025

Read SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimization by Arya Tschand, Muhammad Awad, Ryan Swann, Kesavan Ramakrishnan, Jeffrey Ma, Keith Lowery, Ganesh Dasika, Vijay Janapa Reddi, [6pt] Harvard University, [6pt] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20258.txt.

Introduction

Arya Tschand and seven co-authors published a paper in 2025 on using large language models to speed up graphics chip code the way human experts do. Expert tuners win speed by matching a program's memory pattern to the chip's layout, a craft that takes weeks per kernel. Search-based AI tools tune runtimes blindly, without that hardware sense.

Their system feeds the model the workload's access patterns, chip specifications, profiling logs, and notes on past attempts. For one classic matrix kernel it found in under five minutes the arrangement that took experts two weeks. Across ten varied kernels it produced winning layouts for nine, with speedups up to about twofold. The work points toward AI engineers that reason about hardware, not just code.

Catalog record

CREATORArya Tschand, Muhammad Awad, Ryan Swann, Kesavan Ramakrishnan, Jeffrey Ma, Keith Lowery, Ganesh Dasika, Vijay Janapa Reddi, [6pt] Harvard University, [6pt] [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SwizzlePerf: Hardware-Aware LLMs for GPU Kernel Performance Optimization Arya Tschand, Muhammad Awad, Ryan Swann, Kesavan Ramakrishnan, Jeffrey Ma, Keith Lowery, Ganesh Dasika, Vijay Janapa Reddi, [6pt] Harvard University, [6pt] [email protected] Abstract Large language models (LLMs) have shown progress in GPU kernel performance engineering using inefficient search-based methods that optimize around runtime. Any existing approach lacks a key characteristic that human performance engineers rely on for near-optimal utilization – hardware-awareness . By leveraging the workload’s specific memory access patterns, architecture specifications, filtered profiling logs, and reflections on historical performance, we can make software-level optimizations that are tailored to the underlying hardware. SwizzlePerf automatically generates spatial optimizations for GPU kernels on disaggregated architectures by giving LLMs explicit hardware-awareness. For a GEMM kernel, SwizzlePerf takes less than 5 minutes to generate the same hardware-specific optimal swizzling pattern that took expert performance engineers 2 weeks to find. On a suite of 10 diverse ML and Science kernels, SwizzlePerf can generate swizzling patterns for 9 of the kernels that achieve up to a 2.06 × 2.06\times speedup and 70 % 70\% improvement in L2 hit rate. This work is the first of many steps toward systematically creating hardware-aware LLM performance engineering agents. 1 Introduction GPU code performance engineering is a necessary step in enabling efficient machine learning (ML) systems and High-Performance Computing (HPC) applications. Performance engineering requires hardware-software codesign through understanding how the specific workload is executing on the specific underlying hardware. In this work, we automate this process for GPU kernels by imitating this hardware-software codesign process that human performance engineers follow. This is accomplished by giving LLMs hardware-aware context, which unlocks their ability to structure GPU code optimizations around the underlying hardware architecture and scheduling methodology. SwizzlePerf is our proposed hardware-aware LLM workflow that automatically generates swizzling patterns. Swizzling is a transformation that reorders the mapping between data or work and their execution/storage locations to enhance spatial/temporal locality and align with hardware topology. Compilers and runtimes use swizzling for data layouts and for clustering cooperating blocks. In the case study that we evaluate in this paper, swizzling is a programmer-defined remapping of GPU workgroup program IDs (PIDs) that co-locates related tiles on the same accelerator complex die (XCD) to increase per-XCD L2 reuse on GPUs with disaggregated architectures. By default, workgroups are assigned in a round-robin manner across XCDs (Figure 1(b) and Appendix A.2 ). For applications with predictable memory-access patterns, statically arranging computation and data to maximize local L2 access yields substantial benefits. It takes expert GPU performance engineers multiple weeks to make targeted spatial optimizations. SwizzlePerf autonomously finds these optimizations for a wide range of kernels in minutes. Contributions of this work: 1. We facilitate hardware-aware kernel optimization by intentionally crafting context with necessary hardware and scheduling information from profilers and runtimes. 2. We augment IntelliPerf, an open-source autonomous performance engineering tool, with hardware-awareness and evaluate a case study on generating GPU kernel swizzling patterns. 3. We show performance results on a wide range of ML and scientific GPU kernels of up to a 2.1x speedup and 70% L2 hit rate improvement, indicating that hardware-awareness is necessary to unlock hardware-specific optimizations. (a) Optimization loop methodology. (b) GEMM swizzling. Figure 1: SwizzlePerf methodology and example swizzling outcome. (a) The optimization loop begins with the CodeGen LLM call input , which integrates the parsed context of bottleneck metrics, GPU characteristics, and the scheduling policy. The LLM reflects on past bottlenecks and proposes a new swizzling formula in the CodeGen LLM call output . (b) SwizzlePerf generates this pattern for GEMM on a 4-XCD architecture. The swizzling pattern improves locality in the L2 cache by co-locating tiles that reuse rows in A on the same XCD. While the goal is always to improve intra-XCD locality, the exact swizzling formula varies by algorithm. 2 Related Work Prior work on autonomous kernel optimization largely optimizes for runtime as the single objective using three families of methods: (i) heuristic autotuners and analytic cost models that apply rule-based schedules (tiling, unrolling, coalescing) [ 27 , 13 , 6 ] (ii) learned cost models that predict runtime from code or schedule features and drive combinatorial search [ 9 , 28 , 1 ] and (iii) LLM/RL-driven search that performs test-time exploration or multi-turn refinement, ranging from competitive-programming systems [ 18 , 23 ] to GPU-kernel frameworks [ 20 , 8 , 5 , 17 ] . These approaches can work for software-level bottlenecks, but they typically abstract away concrete architectural constraints (cache topology, chiplet boundaries, block-placement policy) and provide minimal task-specific machine feedback. Without hardware-awareness, the search space becomes too broad and the objective (runtime) is noisy due to launch overheads, overlapping bottlenecks, and compute saturation. To converge on hardware-specific optimizations, the LLM needs relevant context that enables hardware-awareness . We approach this problem by imitating human performance engineers: profile the workload, isolate the bottleneck, apply a targeted fix, and precisely evaluate the improvement. Practically, this means intentionally supplying the profiling, architectural, and scheduling context that is relevant to the optimization task and guiding feedback with a bottleneck metric that provides a stable signal aligned with the transformation. By elevating this metric to the core optimization objective, we narrow the search to changes that directly target the bottleneck. For the cache efficiency bottleneck case study in this paper, we measure L2 hit rate as a direct, low-noise proxy metric for spatial locality. To our knowledge, SwizzlePerf is the only work that adds rich context from a suite of profilers into the context to directly reflect cache-locality improvements and improve LLM optimization. Within this framing, we implement swizzling patterns for disaggregated GPUs (e.g., AMD Instinct TM MI300x [ 24 ] ) where multiple accelerator complex dies (XCDs) each host a dedicated L2 cache. By default, workgroups are scheduled round-robin across XCDs (Figure 1(b) , Appendix A.2 ). However, for applications with predictable access patterns, statically co-locating cooperating tiles on the same XCD with swizzling can substantially improve data reuse in the shared L2 cache. Our contribution adds hardware-awareness and optimizes around a bottleneck metric so the search is focused, the signal is stable, and the resulting transformations align with the underlying hardware. 3 Methodology SwizzlePerf (Figure 1(a) ) is a hardware-aware, bottleneck-driven optimization loop. (1) CodeGen LLM Call. We formulate a targeted code-generation request that fixes the optimization objective and explicitly defines the bottleneck metric of interest. The prompt (Appendix A.1 ) bundles a short memory-locality summary of the kernel, a compact trace of prior attempts, and architecture details (e.g., number of XCDs, cache sizes, and the block-scheduling policy). We build on the open source IntelliPerf [ 7 ] framework that runs a hardware-unaware GPU kernel optimization loop for a single bottleneck. We constrain the goal to a specific, hardware-aware transformation. (2) Parsed Context. We construct a structured context from public profilers and documentation. From rocprofv3 [ 4 ] we extract bottleneck metrics, from HIP device attributes [ 3 ] we gather GPU/XCD/cache parameters, and from architecture guides we derive the default block-scheduling policy. This parsed context exposes the metric of interest (for swizzling, this metric is L2 hit rate) and the spatial constraints the remapping must satisfy. Related work does not include any profiler-provided information in context, and IntelliPerf

Read the complete text