Compiler-R1: Towards Agentic Compiler Auto-tuning with Reinforcement Learning

Haolin Pan , Hongyu Lin , Haoran Luo , Yang Liu, Kaichun Yao, Libo Zhang, Mingjie Xing , Yanjun Wu · 2025

Read Compiler-R1: Towards Agentic Compiler Auto-tuning with Reinforcement Learning by Haolin Pan , Hongyu Lin , Haoran Luo , Yang Liu, Kaichun Yao, Libo Zhang, Mingjie Xing , Yanjun Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15701.txt.

Introduction

Haolin Pan and seven co-authors trained a language model agent to tune compilers by trial and reward, in a paper published in 2025. Compiler tuning picks pass sequences that shrink code. Earlier model attempts lacked good reasoning traces to learn from and had thin interaction with the build. Their framework pairs a hand-made reasoning set with two-stage reward training.

The agent learns to probe builds and keep what pays, scored by final size. Across seven sets it cut code counts by more than eight points past the standard tuning flag. The team releases code and data. They read the gain as proof that reward-trained agents can do craft work, not just chat. Compiler work gains a tireless tuner.

Catalog record

CREATORHaolin Pan , Hongyu Lin , Haoran Luo , Yang Liu, Kaichun Yao, Libo Zhang, Mingjie Xing , Yanjun Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15701

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Compiler-R1: Towards Agentic Compiler Auto-tuning with Reinforcement Learning Haolin Pan , Hongyu Lin , Haoran Luo , Yang Liu, Kaichun Yao, Libo Zhang, Mingjie Xing , Yanjun Wu Abstract Compiler auto-tuning optimizes pass sequences to improve performance metrics such as Intermediate Representation (IR) instruction count. Although recent advances leveraging Large Language Models (LLMs) have shown promise in automating compiler tuning, two significant challenges still remain: the absence of high-quality reasoning datasets for agents training, and limited effective interactions with the compilation environment. In this work, we introduce Compiler-R1, the first reinforcement learning (RL)-driven framework specifically augmenting LLM capabilities for compiler auto-tuning. Compiler-R1 features a curated, high-quality reasoning dataset and a novel two-stage end-to-end RL training pipeline, enabling efficient environment exploration and learning through an outcome-based reward. Extensive experiments across seven datasets demonstrate Compiler-R1 achieving an average 8.46% IR instruction count reduction compared to opt -Oz , showcasing the strong potential of RL-trained LLMs for compiler optimization. Our code and datasets are publicly available at https://github.com/Panhaolin2001/Compiler-R1 . 1 Introduction Figure 1: Overview of compiler auto-tuning task. Compiler auto-tuning ashouri2018surveyauto-tuning ; bodin1998iterative ; BOCA focus on automatically select and order compilation passes, modular optimization or analysis steps in modern compilers like LLVM lattner2004llvm , to improve program performance. A typical tuning objective is reducing the Intermediate Representation (IR) instruction count while preserving program correctness, as depicted in Figure 1 . However, this task presents inherent challenging due to the combinatorial explosion of possible pass sequences and the intricate interactions between individual passes. Recent years have witnessed significant advancements in compiler auto-tuning methods, evolving through three major generations: Heuristic-based Methods : Early work in iterative compilation tpe ; RIO ; GA ; opentuner establish fundamental search strategies such as genetic algorithm and simulated annealing. Machine Learning (ML)-enhanced Methods : Subsequent approaches BOCA ; Comptuner ; autophase ; cfsat leveraged ML models to improve tuning efficiency on unseen benchmarks. However, these traditional approaches, including both heuristic and ML methods, often suffer from inherent inefficiency and limited generalization. Large Language Models (LLMs)-based Methods : With the advent of LLMs, recent efforts italiano2024finding ; cummins2024meta ; cummins2023large have explored their application to compiler auto-tuning tasks, mitigating some of the aforementioned limitations. Although recent LLMs DeepSeek-R1 ; jaech2024openai-o1 exhibit remarkable reasoning and problem-solving capabilities, directly applying them to compiler auto-tuning faces two critical challenges. First, there is an absence of high-quality training datasets tailored for auto-tuning . Effective interactive learning for LLM agents requires datasets that include Chain-of-Thought (CoT) reasoning, external tool integration, and environment feedback, which are currently lacking in compiler tuning domains. Second, existing methods based on Supervised Fine-Tuning (SFT) exhibit limited interaction with the compilation environment , restricting the model’s ability to autonomously explore and adaptively learn from feedback, thus resulting in poor generalization to unseen programs. To address these challenges, we propose Compiler-R1 , the first reinforcement learning (RL)-based framework specifically designed to enhance LLM capabilities in compiler auto-tuning. Compiler-R1 introduces two main innovations: (1) A high-quality auto-tuning reasoning dataset: We curate a comprehensive dataset comprising 19,603 carefully constructed samples, encompassing diverse and representative tuning scenarios, enabling effective interactive learning for compiler optimization tasks. (2) A two-stage end-to-end RL training framework: We develop a novel RL framework that equips LLM agents with tool interfaces for autonomous environment exploration and leverages an outcome-based reward function to guide learning. This approach enables the agent to capture both individual pass characteristics and their compositional interactions, significantly enhancing generalization capability and tuning efficiency. We validate Compiler-R1 using various open-source LLM backbones on seven representative benchmarks. Experimental results indicate that Compiler-R1 consistently achieves an average IR instruction count reduction of 8.46% compared to the baseline method ( opt -Oz ), demonstrating the considerable promise of RL-trained LLMs for advancing compiler auto-tuning research. Our principal contributions are summarized as follows: • We construct a comprehensive, high-quality auto-tuning reasoning dataset by rigorously selecting and integrating multiple authoritative compiler datasets, encompassing diverse and representative tuning scenarios. • We propose Compiler-R1 , the first RL-based two-stage training framework specifically designed to augment LLMs with tool-invocation and advanced reasoning capabilities tailored to compiler auto-tuning tasks. • We demonstrate through comprehensive evaluations that Compiler-R1 consistently outperforms other baseline methods, highlighting the substantial promise of RL-enhanced LLM approaches for advancing compiler optimization. 2 Related Work 2.1 Rule-based Reinforcement Learning with Large Language Models Reinforcement Learning (RL) enables agents to perform sequential decision-making by interacting with environments to maximize cumulative rewards. Recent advancements in rule-based RL frameworks, such as DeepSeek-R1 DeepSeek-R1 , have notably enhanced the reasoning and decision-making capabilities of Large Language Models (LLMs). This approach has effectively been extended to various domains, including knowledge retrieval Search-R1 , logical reasoning logic-rl , mathematical problem-solving DeepSeekMath , code generation code-r1 , and GUI action prediction UI-R1 . Nevertheless, the application of rule-based RL to compiler auto-tuning tasks remains largely unexplored. 2.2 Compiler Auto-tuning Compiler auto-tuning aims to automatically identify optimal pass sequences to enhance program performance. Traditional compilers like GCC and LLVM utilize fixed optimization pipelines designed by domain experts, which often do not adapt well to specific programs or hardware. Iterative compilation methods RIO ; GA employing heuristics such as genetic algorithms incur substantial overhead. Supervised learning approaches, like coreset-NVP coreset , rely heavily on labeled optimal sequences, which are challenging to obtain. RL-based methods, such as AutoPhase autophase , dynamically adjust optimization strategies but have limited generalization. Recently, LLM-based approaches cummins2023large have shown potential by directly generating pass sequences from source code; however, they typically lack sufficient environmental interaction, constraining their reasoning and adaptability. 3 Methodology: Compiler-R1 We present Compiler-R1, a novel framework that introduces large language model (LLM)-driven compiler auto-tuning, incorporating both supervised learning and reinforcement learning (RL) techniques. As illustrated in Figure 2 , Compiler-R1 represents the first application of reinforcement learning to LLM-based compiler optimization, enabling models to move beyond imitation and autonomously learn optimization strategies through structured interaction with a simulated compilation environment. Figure 2: Compiler-R1: A two-stage LLM training framework for compiler auto-tuning. Stage 1 (SFT): A foundation dataset derived from synergistic pass analysis is used to create SFT samples. These samples guide the LLM through a CoT process, teaching interaction with instrcount and find_best_pass_sequence (FBPS) tools. Stage 2 (RL): The SFT-initialized LLM agent interacts with the compiler autotuning environment, receiving state feedback and rewards to learn an optimal policy for pass sequence generation. 3.1 Foundation Dataset Construction To enable effective compiler auto-tuning, we construct a high-quality dataset tailored for LLM training, derived from empirical compiler tuning workflows (e.g., CFSAT cfsat ) and enhanced with structural insights. Our process involves the following pipelines: Raw LLVM IR Dataset Preparation. We aggregate multiple public datasets from CompilerGym compilergym , applying coreset-inspired sampling coreset to ensure representational diversity. Programs with more than 10,000 IR instructions are excluded for computational feasibility. The training set comprises six uncurated datasets, while four curated benchmarks (cbench-v1, chstone-v0, mibench-v1, and npb-v0 cbench ; mibench ; npb ; chstone ) are served for out-of-distribution evaluation. Synergy Pass Pair Identification. For each training program P , we derived its high-quality pass sequences S_{P} by collecting synergistic optimization pass pairs (A,B)\in O\times O satisfying the following criteria: \begin{split}S_{P}=\{(A,B)\mid{}&(\operatorname{Count}(\operatorname{Apply}(P,B))<\operatorname{Count}(P))\land\\ &(\operatorname{Count}(\operatorname{Apply}(P,A,B))<\operatorname{Count}(\operatorname{Apply}(P,B)))\}\end{split} (1) where \operatorname{Apply}(P,B) denotes applying pass B to program P ,

Read the complete text