ATiM: Autotuning Tensor Programs for Processing-in-DRAM
Yongwon Shin, Dookyung Kang, Hyojin Sung · 2024
Read ATiM: Autotuning Tensor Programs for Processing-in-DRAM by Yongwon Shin, Dookyung Kang, Hyojin Sung (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19630.txt.
Introduction
Computers that compute inside their own memory promise relief for data-hungry programs like large language models, but programming them is painful. Libraries are hand-tuned, abstractions are thin, and no systematic optimizer exists. The authors present ATiM, a tensor compiler for the UPMEM processing-in-DRAM system that tunes itself. It searches the joint space of host and kernel programs automatically. Special optimizations handle the boundary conditions that this hardware makes awkward, and improved search algorithms cope with the enlarged space of choices.
On real UPMEM hardware, the compiler speeds benchmark kernels by up to 6.18 times. Layers of the GPT-J language model run up to 8.21 times faster than with the stock approach. The authors claim the first fully automated, autotuning code generation for a DRAM-based processing system. They frame ATiM as the bridge between high-level tensor programs and the low-level demands of computing in memory.
Catalog record
| CREATOR | Yongwon Shin, Dookyung Kang, Hyojin Sung |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-19630
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ATiM: Autotuning Tensor Programs for Processing-in-DRAM Yongwon Shin, Dookyung Kang, Hyojin Sung Abstract . Processing-in-DRAM (DRAM-PIM) has emerged as a promising technology for accelerating memory-intensive operations in modern applications, such as Large Language Models (LLMs). Despite its potential, current software stacks for DRAM-PIM face significant challenges, including reliance on hand-tuned libraries that hinder programmability, limited support for high-level abstractions, and the lack of systematic optimization frameworks. To address these limitations, we present ATiM, a search-based optimizing tensor compiler for UPMEM. Key features of ATiM include: (1) automated searches of the joint search space for host and kernel tensor programs, (2) PIM-aware optimizations for efficiently handling boundary conditions, and (3) improved search algorithms for the expanded search space of UPMEM systems. Our experimental results on UPMEM hardware demonstrate performance gains of up to 6.18 × \times for various UPMEM benchmark kernels and 8.21 × \times for GPT-J layers. To the best of our knowledge, ATiM is the first tensor compiler to provide fully automated, autotuning-integrated code generation support for a DRAM-PIM system. By bridging the gap between high-level tensor computation abstractions and low-level hardware-specific requirements, ATiM establishes a foundation for advancing DRAM-PIM programmability and enabling streamlined optimization. Conference: Proceedings of the 52nd Annual International Symposium on Computer Architecture; June 21–25, 2025; Tokyo, Japan Proceedings of the 52nd Annual International Symposium on Computer Architecture (ISCA ’25), June 21–25, 2025, Tokyo, Japan DOI: 10.1145/3695053.3731096 ISBN: 979-8-4007-1261-6/2025/06 † † cc-license: by-sa 1. Introduction Modern applications are increasingly data-driven, leveraging vast amounts of data to optimize processes and improve efficiency. This has led to the inevitable shift of performance bottlenecks from compute unit efficiency to limited memory performance in computing system design. Among various hardware and software-based approaches to continue performance scaling beyond the “memory wall”, processing-in-DRAM (DRAM-PIM) confronts the challenge by placing compute units near memory banks and computing directly on data in the memory instead of moving them. Recently, many hardware and memory vendors presented commercial and prototype DRAM-PIM products ( Devaux, 2019a ; Kim et al., 2021 ; Lee et al., 2022 ) to showcase their potential, providing up to 16 \times higher in-memory compute bandwidth without data transfer overheads. For example, DDR4-based UPMEM ( Devaux, 2019a ) includes a RISC core called DPU (Data Processing Unit) per memory bank, allowing it to fully utilize the internal memory bandwidth and bank-level parallelism for accelerating memory-intensive workloads such as database systems, genomic analysis, and machine learning/deep learning inferences ( Chen et al., 2023b ; Bernhardt et al., 2023 ; Lim et al., 2023 ; Das et al., 2022 ; Gómez-Luna et al., 2023 ; Gogineni et al., 2024 ) . For DRAM-PIM to evolve into a mainstream architectural player, a fully functioning, end-to-end software stack that provides user interfaces and code generation support for programmability and multi-level optimizations for performance is crucial. However, current DRAM-PIM software stacks are still in an early and experimental phase, focusing on supporting a narrow set of target workloads with performance advantages over CPU/GPU. They provide a limited set of heavily hand-tuned libraries ( Kwon et al., 2022 ; Lee et al., 2021 ) for memory-intensive, PIM-friendly operations (e.g., GEMV) based on low-level programming model and compiler support ( Gómez-Luna et al., 2022 ; Chen et al., 2023a ) . Without high-level abstraction layers bridging the gap between such operations and different DRAM-PIM backends (e.g., C programming for UPMEM), substantial programming and engineering efforts must be repeated. Recent research leveraged modern tensor-oriented intermediate representations (IRs) and compilers ( Ragan-Kelley et al., 2013 ; Vasilache et al., 2018 ; Lattner et al., 2021 ) to provide such abstractions. In tensor compilers, multi-dimensional arrays, i.e., tensors, are the base data type, and tensor IR instructions define operations repeated on tensor elements. Thus, tensor IRs can seamlessly represent in-memory computations as nested loops over tensors stored in the memory, facilitating lowering from high-level tensor operations to target-specific instructions. Prior work leveraged tensor IRs to generate codes for different types of PIM hardware ( Khan et al., 2023 ) or determine feasible mappings for tensor data to memory subarrays ( Drebes et al., 2020 ) , focusing on improving the programmability of the PIM software stack. Our key insight is that the true value of tensor compilation for DRAM-PIM lies in the unique optimization opportunities provided by domain-specific tensor abstractions: specifically, enabling effective search-based optimization, i.e., autotuning , and facilitating aggressive transformations that exploit high-level semantic knowledge and hardware-specific constraints during tensor IR lowering. First, the autotuning framework can closely interface with the DRAM-PIM code generator to search the space of host and kernel tensor programs and optimize them simultaneously. Autotuning formulates optimizations as a search problem, exploring different optimized versions of a given tensor operation to identify the best-performing candidate through hardware measurements ( Chen et al., 2018a ; Chen et al., 2018b ; Zheng et al., 2020 ; Feng et al., 2023 ) . This approach can automatically generate codes optimized explicitly for target hardware, often outperforming expert-tuned libraries, thereby providing both programmability and performance benefits. Existing tensor compilers have shown that the autotuning framework can effectively optimize tensor kernel functions on CPUs and GPUs by sampling candidates with varying loop structures and optimization parameters such as loop tiling and unroll factors ( Zheng et al., 2020 ; Feng et al., 2023 ) . While CPU and GPU kernel functions can be optimized independently, UPMEM kernel optimization depends on host-code parameters as the host controls how to distribute tensors across memory banks. This interdependency creates a larger and more complex search space, making the autotuning approach an even more compelling solution for optimal code generation for DRAM-PIM. In addition, tensor compilers can leverage target loop structures and DRAM-PIM hardware features for aggressive, correctness-guaranteed high-level optimizations. Specializing optimization passes for specific operations and hardware can enable optimizations that would be infeasible in low-level compilers without semantic context. These PIM-aware optimizations complement autotuning and low-level approaches, offering unique performance benefits. In this paper, we present ATiM, an optimizing tensor compiler for modern DRAM-PIM. ATiM extends the autotuning framework ( Shao et al., 2022 ) and tensor-level IR lowering and optimization support ( Feng et al., 2023 ) in the Apache TVM compiler ( Chen et al., 2018a ) to provide fully automated search-based code generation for UPMEM ( Devaux, 2019a ; Devaux, 2019b ) . ATiM expands the autotuner to sample candidates from a joint search space including host-to-DPU data distribution strategies and kernel loop-level optimization parameters. While TVM schedule primitives are originally designed for loop transformations in kernel loops, ATiM repurposes them to simultaneously optimize UPMEM host and kernel operations. It also provides the lowering passes to translate these primitives to loop-based tensor IR programs. Tightly integrating autotuning with code generation, in turn, enables ATiM to handle the larger and more complex search space and reduce the evolutionary search overheads. It refines the evolutionary search mechanism and filters infeasible candidates early in the autotuning process by systematically leveraging UPMEM hardware constraints and parallel execution behaviors. In addition, ATiM implements tensor-level code optimizations to aggressively reduce boundary check overheads, exploiting target algorithm and hardware knowledge. Experimental results show that ATiM-compiled kernels outperformed hand-tuned UPMEM libraries ( Gómez-Luna et al., 2022 ; Chen