From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graph
Junfeng Gong Zhiyi Wei Junying Chen Cheng Liu Huawei Li · 2025
Read From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graph by Junfeng Gong Zhiyi Wei Junying Chen Cheng Liu Huawei Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19873.txt.
Introduction
Junfeng Gong and four co-authors help small language models write optimized GPU code like large ones, without sending code to the cloud. Big models generate good CUDA, but cloud APIs risk leaking code and local deployment is costly; small models run privately yet reason too weakly for complex optimization. Their framework, ReGraphT, needs no additional training.
ReGraphT stores recorded optimization sessions as a reasoning graph, where combined optimizations become transitions between states, and searches that graph with Monte Carlo methods to guide the small model. A tiered CUDA benchmark supports the evaluation. On two benchmark suites the method averaged a speedup of about two and a third times, beating fine-tuned specialist models and other retrieval approaches and bringing small models close to large-model quality.
Catalog record
| CREATOR | Junfeng Gong Zhiyi Wei Junying Chen Cheng Liu Huawei Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19873
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Large to Small: Transferring CUDA Optimization Expertise via Reasoning Graph Junfeng Gong Zhiyi Wei Junying Chen Cheng Liu Huawei Li Abstract Despite significant evolution of CUDA programming and domain-specific libraries, effectively utilizing GPUs with massively parallel engines remains difficult. Large language models (LLMs) show strong potential in generating optimized CUDA code from sequential code. However, using LLMs in practice faces two major challenges: cloud-based APIs pose risks of code leakage, and local deployment is often computationally expensive and inefficient. These drawbacks have spurred interest in small language models (SLMs), which are more lightweight and privacy-friendly. Encouragingly, recent studies show that SLMs can achieve performance comparable to LLMs on specific tasks. While SLMs can match LLMs on domain-specific tasks, their limited reasoning abilities lead to suboptimal performance in complex CUDA generation according to our experiments. To bridge this gap, we propose ReGraphT, a training-free, retrieval-augmented generation framework that transfers LLM-level reasoning to smaller models. ReGraphT organizes CUDA optimization trajectories into a structured reasoning graph, modeling the combined CUDA optimizations as state transitions, and leverages Monte Carlo Graph Search (MCGS) for efficient exploration. We also present a CUDA-specific benchmark with difficulty tiers defined by reasoning complexity to evaluate models more comprehensively. Experiments show that ReGraphT outperforms HPC-specific fine-tuned models and other retrieval-augmented approaches, achieving an average 2.33× speedup on CUDAEval and ParEval. When paired with DeepSeek-Coder-V2-Lite-Instruct and Qwen2.5-Coder-7B-Instruct, ReGraphT enables SLMs to approach LLM-level performance without the associated privacy risks or excessive computing overhead. 1 Introduction The continuous performance improvement of NVIDIA GPUs ( Dally et al. 2021 ; Lindholm et al. 2008 ; Nickolls & Dally 2010 ; Owens et al. 2008 ) has solidified CUDA as a dominant programming model for high-performance computing tasks, including AI and scientific computing. However, writing efficient CUDA code that fully exploits the massively parallel processing capabilities of GPUs remains a significant challenge. To alleviate the burden of CUDA programming, prior research has proposed domain-specific libraries, programming frameworks, and even domain-specific languages ( Brahmakshatriya & Amarasinghe 2022 ; Tillet et al. 2019 ; Chen et al. 2018 ; Che et al. 2008 ; Bell & Hoberock 2012 ; Hong et al. 2019 ) . While these approaches significantly enhance productivity and deliver competitive performance, they often demand substantial engineering effort, are restricted to specific application domains, and suffer from compatibility issues with frequent NVIDIA software updates. Recently, large language models (LLMs) have shown remarkable potential in code generation tasks ( Qiu et al. 2020 ) across a wide range of programming languages—including Python, C/C++, Verilog, and even high-level synthesis code for FPGAs—demonstrating new opportunities for automatic CUDA code generation from sequential code ( Li et al. 2023 ; Rozière et al. 2024 ; Luo et al. 2023 ; Zheng et al. 2023 ; Zhang et al. 2024 ) . Encouraging progress has already been observed in this direction ( Bendi-Ouis et al. 2025 ; Yan et al. 2024 ; Miranda et al. 2025 ) . Nevertheless, deploying LLMs such as DeepSeek locally is highly resource-intensive due to their large-scale architecture. On the other hand, using cloud-based APIs raises concerns over potential code leakage and privacy violations. These limitations have fueled interest in small language models (SLMs), which are significantly more lightweight, support convenient local deployment, and mitigate privacy risks. Notably, recent studies have shown that SLMs can achieve performance on par with LLMs in certain domain-specific code generation tasks ( Brown et al. 2020b ) . Despite this promise, training SLMs from scratch remains extremely challenging due to limited training data and convergence difficulties. Consequently, fine‐tuning has emerged as a practical means to produce compact, domain‐specialized, and compute‐efficient SLMs. For example, HPC‑Coder‑V1 and V2 leverage curated parallel‐code datasets to fine‐tune large LLMs and substantially improve their ability to generate high‐performance parallel programs ( Nichols et al. 2024b ; Chaturvedi et al. 2025 ) , while RLPF employs reinforcement learning to further align LLM outputs with performance objectives ( Nichols et al. 2024c ) . However, we find that the efficacy of these fine‐tuned SLMs degrades markedly on problems demanding deeper, multi‐step reasoning. To quantify this effect, we sampled 20 benchmarks from ParEval and applied chain‑of‑thought ( Wei et al. 2023 ) (CoT) prompting to both the 671B DeepSeek-R1 model and smaller 7B/14B code‐specific SLMs. Figure 1 reports each model’s average number of reasoning steps alongside the performance of the generated code. The results reveal that, while SLMs match LLMs on simpler tasks, they take significantly fewer reasoning steps and yield lower code quality on more complex benchmarks. This gap underscores the need for new techniques that can extend the reasoning capacity of lightweight models without sacrificing their deployment advantages. In addition to fine-tuning, retrieval-augmented generation (RAG) is another widely adopted strategy for enhancing SLM performance by injecting external information directly into the model’s context. Prior works such as EVOR and Repoformer ( Su et al. 2024 ; Wu et al. 2024 ) have successfully applied RAG to general code generation tasks, demonstrating notable improvements in output quality, especially for code involving recurring patterns or known structures. However, while RAG effectively enriches contextual knowledge, it does not directly improve the model’s reasoning capabilities. As a result, RAG-enhanced SLMs still struggle with generation tasks that require multi-step logical reasoning, leaving a critical gap in handling more complex coding problems. Figure 1: Average number of reasoning steps and the performance of the generated code with SLMs and LLMs. To enhance the reasoning capabilities of SLMs in CUDA code generation, we propose ReGraphT, a training-free framework that augments SLMs with a structured reasoning process of CUDA-specific optimizations. ReGraphT leverages the reasoning strength of LLMs to collect step-by-step CUDA optimization trajectories, which are then aggregated into a unified CUDA reasoning graph. This graph captures the intermediate states and transitions involved in transforming sequential code into efficient CUDA implementations. ReGraphT formulates the CUDA code generation task for SLMs as a graph-based reasoning problem and incorporates Monte Carlo Graph Search (MCGS) to guide the search over the graph efficiently. In addition, to support systematic evaluation, we also introduce CUDAEval, a benchmark suite specifically designed to assess CUDA code generation. CUDAEval organizes tasks into multiple difficulty levels based on the complexity of their underlying reasoning trajectories, enabling fine-grained analysis of model performance across different levels of challenge. Our contributions are summarized as follows: • We propose ReGraphT, a novel, training-free framework designed to mitigate the limited reasoning ability of SLMs in CUDA code generation. ReGraphT employs a CUDA Reasoning Graph to encode optimization trajectories extracted from LLMs, thereby enabling SLMs to benefit from the rich multi-step reasoning encoded by larger models. The framework is open sourced on GitHub 1 1 1 https://anonymous.4open.science/r/ReGraphT-1A47 . • We formulate CUDA code generation as a graph-based state transition problem and apply Monte Carlo Graph Search (MCGS) to efficiently navigate the CUDA Reasoning Graph. This formulation enables effective decision-making at each optimization stage, enhancing the quality of generated CUDA code. • We design CUDAEval, a CUDA-specific benchmark suite that categorizes code generation tasks into levels of reasoning difficulty. Experimental results show that ReGraphT significantly improves the reasoning and code generation performance of SLMs, narrowing the gap between lightweight and large models in CUDA optimization tasks. 2 Related