Interleaving Large Language Models for Compiler Testing DOI: 10.1145/3763079 Journal: PACMPL Volume: 9 OOPSLA2 301 10 CCS: Software and its engineering Compilers CCS: Software and its engineering Software testing and debugging

Yunbo Ni, Shaohua Li · 2025

Read Interleaving Large Language Models for Compiler Testing DOI: 10.1145/3763079 Journal: PACMPL Volume: 9 OOPSLA2 301 10 CCS: Software and its engineering Compilers CCS: Software and its engineering Software testing and debugging by Yunbo Ni, Shaohua Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18955.txt.

Introduction

Yunbo Ni and Shaohua Li find compiler bugs by letting language models write small code pieces once, then snapping them together into big test programs. Model-written tests are usually too simple, and calling the model for every test burns money. LegoFuzz splits the work into an offline and an online phase.

Their 2025 paper has the model stockpile small but feature-rich fragments offline. Online, the tool combines the fragments strategically into valid, demanding programs and feeds them to the compilers. Against GCC and LLVM, the two giants of C compilation, the hunt found sixty six bugs. Nearly half were miscompilations, the silent wrong-code class that earlier model-based tools never caught. Reuse, not volume, is the efficiency trick.

Catalog record

CREATORYunbo Ni, Shaohua Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18955

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interleaving Large Language Models for Compiler Testing DOI: 10.1145/3763079 Journal: PACMPL Volume: 9 OOPSLA2 301 10 CCS: Software and its engineering Compilers CCS: Software and its engineering Software testing and debugging Yunbo Ni, Shaohua Li Abstract . Testing compilers with AI models, especially large language models (LLMs), has shown great promise. However, current approaches struggle with two key problems: The generated programs for testing compilers are often too simple, and extensive testing with the LLMs is computationally expensive. In this paper, we propose a novel compiler testing framework that decouples the testing process into two distinct phases: an offline phase and an online phase. In the offline phase, we use LLMs to generate a collection of small but feature-rich code pieces. In the online phase, we reuse these code pieces by strategically combining them to build high-quality and valid test programs, which are then used to test compilers. We implement this idea in a tool, LegoFuzz , for testing C compilers. The results are striking: we found 66 bugs in GCC and LLVM, the most widely used C compilers. Almost half of the bugs are miscompilation bugs, which are serious and hard-to-find bugs that none of the existing LLM-based tools could find. We believe this efficient design opens up new possibilities for using AI models in software testing beyond just C compilers. † † cc-license: by 1. Introduction Compilers play a fundamental role in the modern software ecosystem. However, despite significant efforts to enhance their reliability, they remain prone to bugs ( Chen et al. 2020 ) . Therefore, large-scale compiler testing is essential for identifying and eliminating these bugs. Various approaches have been proposed, such as random program generation ( Yang et al. 2011 ; Livinskii et al. 2020 ; Livinskii et al. 2023 ) and mutation-based testing ( Le et al. 2014 ; Sun et al. 2016 ) . With the advancement of Large Language Models (LLMs), new tools have emerged to enhance compiler testing. A notable example is Fuzz4All ( Xia et al. 2024 ) , a universal fuzzer designed for multiple purposes, including compiler testing. It leverages LLM-generated prompts within the testing loop to iteratively generate test programs, improving the efficiency and diversity of fuzzing. Another tool, WhiteFox ( Yang et al. 2024 ) , is specifically designed for deep learning compilers and features a multi-agent framework. It generates prompts using relevant documentation and example code to guide LLMs in producing targeted and effective test programs. Although current LLM-based compiler testing has demonstrated great potential, its practical deployment comes with notable challenges. In real-world applications, two major concerns arise: (1) the difficulty of ensuring the quality and validity of generated test cases and (2) the high computational cost of integrating LLMs into large-scale testing workflows. Below, we elaborate on the two core challenges. Figure 1 . Examples of LLM-generated erroneous code. ➤ Challenge 1: Low quality of testing programs. In compiler testing, we typically need to generate complex and valid programs, both syntactically and semantically . However, the programs generated by the current LLM-based tools do not always meet these requirements. Unlike natural languages, programs contain rich grammar and semantic constraints, which LLMs are not fully aware of ( Jiang et al. 2024 ) . Thus, LLMs are fundamentally limited in that they are not guaranteed to generate valid programs. For example, Figure 1 (a) shows the C program generated by the model used in Fuzz4All, which contains a syntax error and cannot be compiled by GCC. Figure 1 (b) shows another LLM-generated program, which can be compiled but contains a semantic error, i.e. , a buffer overflow in line 5. Such invalid programs can only be used to find crash bugs, not miscompilation bugs. In fact, neither Fuzz4All nor WhiteFox found any miscompilation bugs in C compilers. All GCC bugs found by Fuzz4All are related to compiler front-end crashes rather than core compiler optimizations. WhiteFox identified only two bugs in LLVM, one involving a compiler backend crash and another related to a crash in error diagnostics. According to its own reports, Fuzz4All generates only 37.26% valid C programs for GCC. The average length of its generated programs is just 18 lines, while WhiteFox produces slightly longer programs with an average of 21 lines based on our evaluation. Hence, how to generate complex ( e.g. , thousands of lines) yet valid programs with LLMs for compiler testing is a challenging problem. ➤ Challenge 2: High computational cost. Currently, using LLMs for fuzzing is expensive, both computationally and financially. An effective fuzzing process is usually required to generate a large volume of test inputs in a short period ( Zhu et al. 2022 ) . However, current LLM-based fuzzers are highly constrained by the sheer amount of computational cost of LLMs. For example, Fuzz4All can generate only around 16,000 valid C programs in 24 hours. Extending these LLM-based fuzzers to produce hundreds of millions of programs would require an extremely large amount of computational or financial resources. In contrast, traditional fuzzers can easily achieve such volumes of data on modern hardware, as there is some code analysis checking the changes or the generated code to some degree. For example, Csmith generates around 1 million test programs overnight, with 99.96% validity rate ( Even-Mendoza et al. 2023 ; Tu et al. 2022 ; Even-Mendoza et al. 2022 ) . ➤ Our core idea. This paper addresses the challenges mentioned above by innovatively decoupling the entire compiler testing process into two distinct phases: an offline phase and an online phase . Rather than relying solely on LLMs throughout the entire pipeline, we integrate both LLMs and traditional program generation/mutation techniques, each assigned to the phase where they are most effective. In the offline phase, we utilize LLMs to generate small yet feature-rich and valid programs, which are then reused as building blocks for the online phase. In the online phase, we iteratively use the building blocks to generate increasingly complex yet valid programs for compiler testing. This hybrid approach balances the strengths of both LLMs and traditional methods, optimizing both the quality and the cost of LLM-based compiler testing. ➤ Our approach: LegoFuzz . We propose LegoFuzz , a novel compiler testing framework that strategically decouples LLM usage into two synergistic phases: an offline phase for collecting high-quality code building blocks through LLM-guided transformation of real-world code, and an online phase for iterative program synthesis that efficiently reuses these building blocks to generate complex test cases without further LLM invocations. Our goal in the first offline phase is to generate programs with measurable complexity while maintaining syntactic and semantic validity. An observation is that LLM-generated code often exhibits recurring structural patterns that correlate with the models’ pre-training data ( Deng et al. 2024 ) . This observation suggests potential limitations in the diversity of generated programs, which could affect their effectiveness for compiler testing. To address this limitation, we leverage existing open-source code as templates to guide LLMs in generating or transforming code snippets. We then validate their syntactic and semantic validity and only keep the valid ones. All code snippets are organized as single functions to facilitate the validation as well as the future synthesis process. Using this

Read the complete text