Benchmarking and Enhancing LLM Agents in Localizing Linux Kernel Bugs
Zhenhao Zhou, Zhuochen Huang, Yike He, Chong Wang, Jiajun Wang, Yijian Wu, Xin Peng, Yiling Lou · 2025
Read Benchmarking and Enhancing LLM Agents in Localizing Linux Kernel Bugs by Zhenhao Zhou, Zhuochen Huang, Yike He, Chong Wang, Jiajun Wang, Yijian Wu, Xin Peng, Yiling Lou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19489.txt.
Introduction
Zhenhao Zhou and co-authors present a 2025 paper on how well AI agents can find faults in the Linux kernel. The kernel underpins billions of devices, and locating the exact buggy code is a hard, high-stakes search. Language model agents shine on smaller benchmarks, so the team built a kernel-scale test from real bugs.
Their benchmark, LinuxFLBench, scores agents on file-level fault location. The best current agent reached only about 42 percent top-one accuracy, far below its scores elsewhere. Huge code size, weak observability, and many confounders explain the struggle. The authors then add an enhancement frame that lifts every agent they tried. The study gives the field a sterner yardstick and a first remedy.
Catalog record
| CREATOR | Zhenhao Zhou, Zhuochen Huang, Yike He, Chong Wang, Jiajun Wang, Yijian Wu, Xin Peng, Yiling Lou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19489
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking and Enhancing LLM Agents in Localizing Linux Kernel Bugs Zhenhao Zhou, Zhuochen Huang, Yike He, Chong Wang, Jiajun Wang, Yijian Wu, Xin Peng, Yiling Lou Abstract The Linux kernel is a critical system, serving as the foundation for numerous systems. Bugs in the Linux kernel can cause serious consequences, affecting billions of users. Fault localization (FL), which aims at identifying the buggy code elements in software, plays an essential role in software quality assurance. While recent LLM agents have achieved promising accuracy in FL on recent benchmarks like SWE-bench, it remains unclear how well these methods perform in the Linux kernel, where FL is much more challenging due to the large-scale code base, limited observability, and diverse impact factors. In this paper, we introduce LinuxFLBench , a FL benchmark constructed from real-world Linux kernel bugs. We conduct an empirical study to assess the performance of state-of-the-art LLM agents on the Linux kernel. Our initial results reveal that existing agents struggle with this task, achieving a best top-1 accuracy of only 41.6% at file level. To address this challenge, we propose LinuxFL + , an enhancement framework designed to improve FL effectiveness of LLM agents for the Linux kernel. LinuxFL + substantially improves the FL accuracy of all studied agents (e.g., 7.2% - 11.2% accuracy increase) with minimal costs. Data and code are available at https://github.com/FudanSELab/LinuxFLBench . 1 Introduction The Linux kernel is a critical system which serves as the foundation for numerous operating systems, servers, and embedded systems. To date, the Linux kernel has undergone continuous evolution over several decades, involving thousands of developers and billions of active users ( linux_kernel_history_report ) . Given the widespread adoption of the Linux kernel, bugs in the Linux kernel can cause serious consequences, affecting a vast number of users. Therefore, extensive research has been dedicated to developing automated software quality assurance techniques (e.g., testing ( bligh2006fully ; DBLP:conf/hpcc/ChenWYZCYM13 ; DBLP:conf/asplos/YangZZ25 ) and debugging ( DBLP:conf/kbse/BissyandeRLM12 ; drgn ; DBLP:conf/usenix/SerranoNTJ0LM20 ; DBLP:conf/eurosys/JeongJLLSK23 ) ) specifically for the Linux kernel. Fault localization (FL), which aims at identifying the buggy code elements (e.g., files or functions) in software, plays a critical role in software quality assurance. Given the codebase of the buggy software and the bug report (e.g., a user-reported bug symptom description), automated FL techniques return a list of buggy code elements ranked by their suspiciousness (i.e., the probability of being buggy). In particular, accurate FL is a prerequisite for bug fixing, as a bug cannot be resolved without correctly identifying the faulty code location. Traditional FL techniques mainly leverage heuristics ( DBLP:conf/prdc/AbreuZG06 ; DBLP:journals/tr/WongDGL14 ) or information retrieval (IR) ( DBLP:conf/icse/ZhouZL12 ; DBLP:conf/kbse/SahaLKP13 ) to identify buggy code elements. More recently, with the advance in large language models (LLMs), LLM agents DBLP:journals/corr/abs-2409-02977 have demonstrated remarkable accuracy in FL. By equipping LLMs with the tool invocation capability, agents can autonomously navigate through the codebase to identify the buggy location. For example, the state-of-the-art agents such as SWE-Agent ( DBLP:journals/corr/abs-2405-15793 ) , AutoCodeRover ( DBLP:conf/issta/0002RFR24 ) , Agentless ( DBLP:journals/corr/abs-2407-01489 ) , achieve around 70% accuracy in localizing buggy files for Python software in the benchmark SWE-bench ( DBLP:conf/iclr/JimenezYWYPPN24 ) . Although achieving promising FL effectiveness, existing agents have been mainly evaluated on general software at moderate scales. It remains unclear how existing agents perform in complex, large-scale software systems like the Linux kernel. In particular, FL in Linux kernel is more challenging than general software due to the following factors. (1) Large-scale Codebase : the Linux kernel has a massive codebase significantly larger than general software. For example, the v5.8 release of Linux kernel includes over 69K files and 28M lines of code ( linux_kernel_history_report ) , which is over 30 times the scale of even the largest project in the most widely-used benchmark SWE-bench. (2) Limited Observability : given the real-time nature of the Linux kernel with the need to minimize overhead, the kernel restricts the use of instrumentation and logging mechanisms during runtime. Additionally, the kernel operates in a privileged mode, isolated from user space. As a result, user-reported bug descriptions often lack detailed runtime information and debugging hints, creating a significant gap between the user description and the actual root causes. (3) Diverse Impact Factors: kernel bugs are influenced by a wide range of factors, including hardware variability (e.g., architectural configurations) and runtime variability (e.g., system load or timing). These factors lead to an exponentially large reasoning space to accurately diagnose the root causes of errors. Therefore, given the unique challenges and the importance of the Linux kernel, this work aims at investigating the FL effectiveness of state-of-the-art LLM agents in the Linux kernel. Benchmark. We first build a new benchmark LinuxFLBench of 250 real-world FL tasks for the Linux kernel. Each FL task in LinuxFLBench includes a user-submitted bug report, the buggy Linux kernel codebase, and the ground-truth buggy locations based on the associated commit patches. LinuxFLBench involves a wide range of Linux kernel bugs, spanning over 120 Linux kernel versions and 66 different kernel components. The FL tasks are significantly more challenging than those in SWE-bench, as evidenced by the substantially larger codebases (10–30× more files and lines of code) and more complex bug reports (approximately 1.5× more words). Empirical Study. On LinuxFLBench , we make the first attempt to evaluate state-of-the-art (SOTA) LLM agents in localizing Linux kernel bugs. Our results reveal the limited FL effectiveness (e.g., 36.8% - 41.6% accuracy) of existing agents in the Linux kernel; moreover, such a FL accuracy is much lower than their performance on general software systems (as seen in SWE-bench), representing a 16.7% - 31.9% accuracy drop. We further perform bad case analysis to find that existing agents mainly miss the buggy files as they fail to capture the related files or to cover complete root causes of kernel bugs. The results indicate that FL in the Linux kernel is indeed a more challenging task, highlighting the need for building more advanced agents to localize bugs in large and complex software systems like the Linux kernel. Technique. Inspired by our study above, we further propose an enhancing framework LinuxFL + , which improves the FL effectiveness of existing agents for the Linux kernel. LinuxFL + incorporates two expansion strategies to refine the prediction results of existing agents: directory-aware expansion to include buggy files based on the repository structure, and potential cause expansion to identify buggy files based on the additional bug knowledge from Linux kernel mailing list (LKML) ( LKML ) . Our evaluation results show that LinuxFL + can substantially improve the FL accuracy of all studied agents (e.g., 7.2% - 11.2% accuracy increase) with minimal costs. Moreover, the ablation analysis confirms the contribution of each expansion strategies. 2 Background and Related Work FL Task Definition. Given the buggy codebase and the bug report, FL techniques identify buggy code elements (e.g., files or functions). Formally, let a codebase be represented as a set of code elements (e.g., files or functions), \mathcal{C}=\{ce_{1},ce_{2},\dots,ce_{N}\} , where N denotes the total number of code elements. A bug report BR typically includes a title, a description, and optional metadata (e.g., component and hardware information in the context of Linux kernel), and can be expressed as