RAG-Based Fuzzing of Cross-Architecture Compilers
Rana Elnaggar, Brian Delgado, and Jason M. Fung · 2025
Read RAG-Based Fuzzing of Cross-Architecture Compilers by Rana Elnaggar, Brian Delgado, and Jason M. Fung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08967.txt.
Introduction
Rana Elnaggar and co-authors published a research paper in 2025 on a language-model tool that fuzzes cross-platform compilers to shake out bugs. Compilers that target many chip types have endless code paths, and hand-writing tests for each path eats expert time. Their tool pairs a language model with a retrieval system stocked with compiler knowledge.
Let loose on Intel's SYCL compiler, the tool drafted tests on its own. Within hours it found eighty-seven cases where code gave wrong answers or crashed the compiler across different chips. The testers needed no prior knowledge of the compiler passes involved. The authors say retrieval-guided fuzzing can give small teams the reach of large ones.
Catalog record
| CREATOR | Rana Elnaggar, Brian Delgado, and Jason M. Fung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08967
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAG-Based Fuzzing of Cross-Architecture Compilers Rana Elnaggar, Brian Delgado, and Jason M. Fung Abstract OneAPI is an open standard that supports cross-architecture software development with minimal effort from developers. It brings DPC++ and C++ compilers which need to be thoroughly tested to verify their correctness, reliability, and security. Compilers have numerous code flows and optimization features. This process requires developers with deep understanding of the different compiler flows to craft testcases specific to target paths in the compiler. This testcase creation is a time-consuming and costly process. In this paper, we propose a large-language model (LLM)-based compiler fuzzing tool that integrates the concept of retrieval-augmented generation (RAG). This tool automates the testcase generation task and relieves experienced compiler developers from investing time to craft testcase generation patterns. We test our proposed approach on the Intel DPC++/C++ compiler. This compiler compiles SYCL code and allows developers to offload it to different architectures, e.g. GPUs and CPUs from different vendors. Using this tool, we managed to identify 87 SYCL code test cases that lead to output value mismatch or compiler runtime errors when compiled using Intel DPC++ and clang++ compilers and run on different architectures. The testcases and the identified unexpected behaviors of the compilers under test were obtained within only few hours with no prior background on the compiler passes under tests. This tool facilitates efficient compiler fuzzing with reduced developer time requirements via the dynamic testcase creation capability provided by an LLM with RAG. I Introduction The Intel DPC++ compiler enables the same source code to be compiled to generate binaries of different vendor architectures. Accomplishing this purpose requires numerous complicated flows that are implemented across several development teams. Thus, this compiler needs to be thoroughly tested to ensure that it does not crash, nor introduce exploitable bugs in the generated binaries or differing behavior on other architectures. Compiler fuzzing has been widely adopted to test different compilers (e.g., LLVM, Nvidia NVCC and OpenCL compilers) [ 1 ] . Prior work has relied on three major paradigms for compiler fuzzing: (1) closed-box fuzzing, which generates random code snippets based on the target language grammar and features of the target language; (2) partially-closed box fuzzing, which utilizes information about test cases coverage to guide further test generation; and (3) open-box fuzzing which generates test cases based on inspecting the source code of the compiler passes. Closed-box and partially closed box fuzzing techniques suffer from low compiler code coverage and the inability to target specific compiler passes. The ability to test a specific compiler pass is crucial during the compiler development phase. Thus, when closed-box and partially closed box fuzzing techniques are applied, multiple bugs remain unrevealed by fuzzing campaigns that continue to run over years [ 2 ] . Open-box fuzzing addresses the limitations in closed and partially-closed fuzzing, however, it utilizes hand-written code generation templates and policies to cover language features related to a specific compiler pass [ 3 ] . Thus, this methodology is highly labor-intensive. It requires experienced compiler developers with extensive knowledge in compiler features and compiler passes to develop these templates and generation policies. In addition, a custom template needs to be implemented for every compiler pass under test. This severely affects the scalability and feasibility of open-box fuzzing. Recently, large language models (LLMs) have proven their success in code generation. The work in [ 1 ] proposes to generate test cases using LLMs with few-shot in-context learning instead of expert-defined code generation templates and policies. They manually select functions within specific compiler passes and then provide the function implementation to LLMs to generate source code to test the target function within the compiler flow. This approach showed superior results on LLVM and deep learning compilers fuzzing. However, when we applied their methodology to the Intel DPC++ compiler without few-shot learning, we have observed that the test cases that are generated do not comply with SYCL language features and suffer from hallucination [ 4 ] and made up code as shown in Fig. 1 . Thus, this approach did not achieve our goal of a fuzzing tool that can be used by non-compiler experts. Figure 1 : Motivation behind using RAG. (a) sample generated code without using RAG; (b) Sample generated code without using RAG The reason behind this observation is that there is plenty of information related to LLVM compiler passes under test in [ 1 ] . Thus, the LLM has been trained on various sources of data covering LLVM optimizations under test. However, in the case of SYCL-specific compiler passes incorporated in Intel DPC++/C++ compiler, they are actively being developed. Thus, there is high possibility that the LLM does not incorporate enough information about the Intel DPC++ compiler to generate correct code. Therefore, in this work we propose to utilize the retrieval augmented generation (RAG) paradigm to provide LLMs with sufficient context that aids it in code generation even when LLMs have not been explicitly trained on Intel DPC++ compiler features and optimization flows. The proposed framework could easily be integrated by compiler engineering teams with the Intel DPC++ compiler development environment to enable continuous fuzzing of the compiler passes as they are being developed. The contributions of this work are listed below. 1. This is the first paper to evaluate utilizing LLMs to fuzz test Intel DPC++ compilers. 2. We extend prior work [ 1 ] by utilizing RAG methodology to generate syntactically and semantically correct sycl code to fuzz test Intel DPC++ compilers. 3. We present an end-to-end flow for RAG-based Intel DPC++ compiler fuzzing that can be extended to other compilers. The rest of the paper is organized as follows. Section II presents related work. Section 3 explains the proposed approach. Section IV presents experimental setup and discusses the results. Section V outlines limitations and future work. Finally, Section VI concludes the paper. II Related Work Figure 2 : Typical compiler fuzzing flow. Compilers are significantly complex software. Fuzzing compilers has been an open research problems for over a decade. Over the years, various methodologies have been proposed to efficiently fuzz compilers. The typical flow for fuzzing compilers is as shown in Fig. 2 . There is a code generation module to generate random code that triggers various compiler components. Differential testing is then performed to compile the generated code using different compiler versions, different compilers or different compiler flags and optimization flags. If the outcome of the execution of the generated binaries is different, a compiler bug is flagged. We classify the compiler fuzzing literature into two categories, the pre-Large language models (pre-LLM) era and post-LLM era. In the pre-LLM era, random code is generated using language-grammar generators [ 5 , 6 , 7 , 8 ] or hand-crafted templates by experienced compiler experts for each compiler component/transformation pass/optimization pass under tests [ 9 , 10 ] . In the post-LLM era, code is generated using LLMs [ 1 ] . II-A Pre-LLM compiler fuzzing The work in [ 5 ] proposes a random test case generation methodology that targets fuzzing multiple C language compilers (e.g., GCC and LLVM, CIL, TCC, and Open64). The random generation of programs in this approach relies on a subset of C language grammar