Tutoring LLM into a Better CUDA Optimizer

Matyáš Brabec, Jiří Klepl, Michal Töpfer, Martin Kruliš · 2025

Read Tutoring LLM into a Better CUDA Optimizer by Matyáš Brabec, Jiří Klepl, Michal Töpfer, Martin Kruliš (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16933.txt.

Introduction

Matyáš Brabec, Jiří Klepl, Michal Töpfer, and Martin Kruliš test AI coding help in a 2025 paper. CUDA is the language used to program graphics processors. Fast CUDA code needs expert knowledge of parallel patterns. The team asked whether top reasoning models can write it well.

Models produced code for well-known tasks, then received graded hints. Automatic checks measured correctness and speed gains. Human reviews added a closer look at code quality. An interactive mode let models fix their own earlier mistakes. The models proved to be skilled coders overall. Still, only with tutoring did they match expert-written speed. The study maps which optimizations models find alone and which need guidance.

Catalog record

CREATORMatyáš Brabec, Jiří Klepl, Michal Töpfer, Martin Kruliš
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16933

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tutoring LLM into a Better CUDA Optimizer Matyáš Brabec, Jiří Klepl, Michal Töpfer, Martin Kruliš Abstract Recent leaps in large language models (LLMs) caused a revolution in programming tools (like GitHub Copilot) that can help with code generation, debugging, and even performance optimization. In this paper, we focus on the capabilities of the most recent reasoning models to generate optimized CUDA code for predefined, well-known tasks. Our objective is to determine which types of code optimizations and parallel patterns the LLMs can perform by themselves and whether they can be improved by tutoring (providing more detailed hints and guidelines in the prompt). The generated solutions were evaluated both automatically (for correctness and speedup) and manually (code reviews) to provide a more detailed perspective. We also tried an interactive approach where the LLM can fix its previous mistakes within a session. The results indicate that LLMs are quite skilled coders; however, they require tutoring to reach optimized solutions provided by parallel computing experts. 1 Introduction Large language models (LLMs) demonstrated impressive abilities to generate and modify code in various programming languages. This development has led to a revolution in programming tools, such as GitHub Copilot, that can help with code generation, debugging, and performance optimization. Recently, reasoning LLMs have been introduced [ 8 ] . Building on the chain of thought [ 6 ] , reasoning models solve the problem step by step by first reasoning about it (producing intermediate output) and then generating the final answer. This improves the results as the LLM can prepare the high-level structure before writing the code. We opted for using OpenAI o3-mini in this work as it is a state-of-the-art among reasoning LLMs at the time of writing. While LLMs show promising capabilities in generating code [ 5 ] , many open questions remain regarding their capabilities in more complex domains like high-performance parallel computing. Although several papers have been published focusing on this topic [ 7 , 4 ] , their approach was mostly quantitative as they tried to map how many assignments from particular technologies (e.g., OpenMP, CUDA, or MPI) LLMs could solve. In this paper, we take a more detailed and narrow approach to study the quality of the generated code and how the quality correlates with the level of detail of the assignment specification. It has been widely suggested that AI might replace junior-level programmers in the foreseeable future, so our experiments were designed with a similar philosophy. We selected three different CUDA assignments based on our experience and tried to guide the LLM in the right optimization decisions — similarly to how a tutor or a teacher would guide a junior programmer or a student. 1.1 Research questions and outline Our main objective is to assess the LLM capabilities in generating optimized CUDA code for well-known tasks to answer the following research questions: Q1: Can LLMs generate well-optimized CUDA code without specific directions? Q2: Does the quality of the code (particularly the selected optimization) improve with more detailed prompts? Q3: Can LLMs follow specific directions about how to optimize particular code? Q4: Is interactive prompting (with tutoring) better than a single prompt with detailed specifications? Q5: Can LLM set proper tuning parameters of an algorithm based on the reasoning about the given code and how it executes on GPU architecture? Section 2 presents the CUDA assignments we have selected for our experiments and their expected solutions. Our prompting techniques and tutoring process are described in Section 3 . Section 4 summarizes the experimental evaluation and reviews of the generated code. Section 5 summarizes the related work, and Section 6 concludes the paper. In addition, we provide a GitHub repository 1 1 1 https://github.com/matyas-brabec/2025-europar-llm where we publish all the prompts, testing scenarios, generated results, evaluation frameworks and results, and helper scripts used in our research. 2 Assignments We chose three well-known assignments for our experiments, so the LLM should have no trouble designing correct solutions. However, they are not as profound as, for instance, matrix multiplication, so the LLM is expected to deduce the optimizations. Furthermore, we have vast experience in these assignments, which allows us to design good tutoring strategies. 2.1 Histogram Computing histograms is a common task in data analysis. A histogram represents the number of occurrences of each value in a dataset. More specifically, we focus on a histogram that counts characters in plain text. As a minor tweak, only a (continuous) sub-range of the ASCII set is collected; other characters are ignored. The algorithm is given an array of characters (a loaded text file) and a range of char values ( from , to ). The result is a histogram of \emph{to}-\emph{from}+1 bins, where bin i corresponds to a character with ASCII value i+\emph{from} . Let us also point out that this assignment is a part of our advanced parallel programming curriculum, and it is used to introduce common CUDA optimizations to the students. Hence, we have vast experience in how to tutor students to achieve the optimal solution. Expected solutions and optimizations: The most straightforward solution ( baseline ) is to process one input character per CUDA thread, check that it is in the from – to range, and increment its corresponding bin using an atomic instruction. Both the input and the histogram are in the global memory. This solution is simple and correct but not very efficient. Since each input character can initiate an atomic write to global memory, having thousands of threads running in parallel with fewer than 256 bins will cause the atomic operations to collide frequently. The first optimization uses shared memory to store a local copy of the histogram. This way, a thread block can aggregate the updates while reducing atomic collisions since only the block updates its local copy. However, the shared memory must be initialized to zeros, and the local histogram must be copied to the global memory at the end of the thread block computation. The effect of shared memory privatization increases with the number of input data processed by one block. Hence, a second optimization is explicitly assigning multiple characters per thread . The final optimization further reduces the number of atomic collisions within a thread block by placing multiple histogram copies in the shared memory. We use one copy per warp lane (thread t uses copy t\bmod 32 ), and each copy must be placed in its own memory bank to avoid bank conflicts. That is achieved by stridden indexing where the value i of a copy c is stored at offset i\cdot 32+c . Let us state that the final optimization exhibits a speedup exceeding two orders of magnitude over the baseline (using a modern Nvidia GPU and having a sufficiently large input). 2.2 Game of Life stencil The Game of Life [ 2 ] (GoL) is a cellular automaton played on a two-dimensional grid where each cell is either alive or dead . The assignment is to compute one iteration of the Game of Life. Given an input grid, the algorithm produces a new grid where each cell state is updated based on its eight immediate neighbors. A live cell survives only if it has two or three live neighbors; otherwise, it dies

Read the complete text