RepoAudit : An Autonomous LLM-Agent for Repository-Level Code Auditing

Jinyao Guo ∗, Chengpeng Wang ∗, Xiangzhe Xu, Zian Su, Xiangyu Zhang · 2025

Read RepoAudit : An Autonomous LLM-Agent for Repository-Level Code Auditing by Jinyao Guo ∗, Chengpeng Wang ∗, Xiangzhe Xu, Zian Su, Xiangyu Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-18160.txt.

Introduction

Jinyao Guo and four co-authors present an AI agent that audits whole code repositories for bugs in a 2025 paper. Language models can review code without compiling it, but auditing large projects strains their limited context and they sometimes invent false bugs. Their agent explores the codebase on demand, tracing how data flows along realistic program paths inside each function.

A separate validator checks the data flow facts and whether the path conditions for each suspected bug can actually hold, cutting false alarms. Across fifteen benchmark projects the agent found forty real bugs with about seventy eight percent precision, at a few dollars and under half an hour per project. It also surfaced many new bugs in famous open source projects, most since confirmed or fixed.

Catalog record

CREATORJinyao Guo ∗, Chengpeng Wang ∗, Xiangzhe Xu, Zian Su, Xiangyu Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-18160

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RepoAudit : An Autonomous LLM-Agent for Repository-Level Code Auditing Jinyao Guo ∗, Chengpeng Wang ∗, Xiangzhe Xu, Zian Su, Xiangyu Zhang Abstract Code auditing is the process of reviewing code with the aim of identifying bugs. Large Language Models (LLMs) have demonstrated promising capabilities for this task without requiring compilation, while also supporting user-friendly customization. However, auditing a code repository with LLMs poses significant challenges: limited context windows and hallucinations can degrade the quality of bug reports, and analyzing large-scale repositories incurs substantial time and token costs, hindering efficiency and scalability. This work introduces an LLM-based agent, RepoAudit , designed to perform autonomous repository-level code auditing. Equipped with agent memory, RepoAudit explores the codebase on demand by analyzing data-flow facts along feasible program paths within individual functions. It further incorporates a validator module to mitigate hallucinations by verifying data-flow facts and checking the satisfiability of path conditions associated with potential bugs, thereby reducing false positives. RepoAudit detects 40 true bugs across 15 real-world benchmark projects with a precision of 78.43%, requiring on average only 0.44 hours and $2.54 per project.Also, it detects 185 new bugs in high-profile projects, among which 174 have been confirmed or fixed. We have open-sourced RepoAudit at https://github.com/PurCL/RepoAudit . † † affiliationnotice: Equal contribution 1 Introduction The rapid innovation of large language models (LLMs) has remarkably enhanced the productivity of software developers ( Wang et al., 2021 ; Rozière et al., 2023 ; Guo et al., 2024 ) . LLM-powered IDE plugins, such as Copilot, enable agile code generation by facilitating flexible interactions with LLMs ( Barke et al., 2023 ) . However, in the era of LLMs, auditing the rapidly expanding codebase presents a more formidable challenge than writing code. Traditional program analysis techniques, such as dynamic and static analysis, focus primarily on observing runtime behaviors or symbolically reasoning about intermediate code generated during compilation ( King, 1976 ; Cadar et al., 2008 ; Calcagno et al., 2009 ; Sui and Xue, 2016 ; Shi et al., 2018a ) . Unfortunately, software systems are often non-executable and even uncompilable during the human-LLM collaborative development phase. Moreover, existing analysis techniques demand substantial expertise, such as a deep understanding of compiler internals ( Zhang et al., 2024 ; Zhou et al., 2024 ) . Therefore, current program analysis approaches often fall short in meeting the practical requirements of code auditing in real-world contexts ( Johnson et al., 2013 ) . In recent years, extensive research has focused on leveraging LLMs for code auditing via prompt engineering ( Fang et al., 2024 ; Hao et al., 2023 ; Ding et al., 2024 ) . Unlike traditional analysis techniques, LLM-driven code auditing directly analyzes source code without program compilation or execution. By describing analysis requirements through natural language and few-shot examples in prompts, the customization of analysis is also significantly simplified. However, many existing LLM-driven code auditing techniques are largely restricted to small-scale codebases, such as smart contracts ( Sun et al., 2024 ; Zhang and Zhang, 2024 ) , and lack the capability to support repository-level code auditing in complex, real-world scenarios. Direct Prompting Hardly Works. A straightforward solution is to break down a repository into smaller pieces and prompt the model with individual pieces. This approach often falls short for non-local bugs, which may require reasoning across a large number of interconnected code snippets spanning multiple functions, classes, and files. Even with significant future advancements in model reasoning capacity, the fundamental differences between programs and natural language texts for which transformer models were designed make LLMs insufficient for comprehensive repository analysis. In particular, a code repository can be conceptualized as an enormous graph, with nodes representing individual statements and edges capturing intricate relationships such as control flow, data flow, and other interdependencies between statements. These relationships are precisely defined and immensely complex at the repository level. For instance, the number of data-flow edges in the project shadowsocks-libev exceeds one million, significantly surpassing the complexity of the implicit graphical structure present in any LLM’s pre-training data sample. Such distribution differences render direct prompting ineffective as shown by our experiments in Section 2.2 . In addition, detecting many types of bugs requires reasoning about properties along specific program paths. For instance, a memory leak occurs when allocated memory is not freed along some program path. Detecting such so-called path-sensitive bugs ( Shi et al., 2018a ) necessitates unfolding the program’s graph structure into individual paths and analyzing these paths one by one. However, this leads to the well-known path explosion problem, as the number of paths grows exponentially with the number of statements. Hence, direct prompting is akin to presenting a large project on an enormous screen and expecting a human auditor to identify bugs along complex and lengthy paths solely by reading and interpreting the code—a task highly unlikely to succeed. Human Auditing. In practice, human auditors do not operate in such a manner. As revealed by existing cognitive science literature ( Anicic et al., 2012 ) , humans are highly effective at reasoning about events that occur in order, e.g., in temporal or spatial order. Human auditors hence tend to explore complex graphical structures in code by following paths that denote the execution order. To avoid excessive exploration, they traverse only a subset of paths most relevant to the targeted property, leveraging implicit abstraction to preclude irrelevant paths. An example of detecting null pointer dereference can be found in Section 2.1 . Figure 1: A simplified code snippet from the project sofa-pbrpc contains a real NPD bug found by RepoAudit . The blue dashed arrows indicate the edges in the call graph. The red solid arrows show the data-flow facts indicating the null value propagation. The call graph of the project contains 1,508 nodes and 6,196 edges, while its data dependence graph contains 160,875 nodes and 360,096 edges. Our Solution. It is widely believed that LLMs operate similarly to humans but with significantly greater “endurance” and access to a broader spectrum of knowledge ( Li et al., 2023 ; Long et al., 2024 ; Qian et al., 2024 ) . Building on this perspective, we propose a novel LLM-based repository auditing agent, named RepoAudit , inspired by human auditing practices. RepoAudit addresses the fundamental misalignment between LLMs’ tendency to reason sequentially and the inherently complex graphical structures of software repositories through path-sensitive and demand-driven graph traversal. By leveraging LLMs’ abstraction capabilities, RepoAudit mitigates path explosion by excluding irrelevant code regions and sub-paths. Also, it minimizes inherent hallucinations by sanitizing final outputs through the validation of several well-formed properties. More specifically, the agent RepoAudit consists of three components, including the initiator, the explorer, and the validator. First, the initiator identifies starting points based on the properties under investigation, such as NULL values when scanning for null pointer dereference bugs. Second, the explorer traverses the relevant functions on demand. Similar to how human auditors analyze code function by function, the explorer starts by querying the LLMs with the functions containing these starting points. Instead of explicitly and programmatically enumerating individual paths inside a function, as in traditional compiler-based automated scanners, the explorer leverages LLMs’ inherent ability to implicitly distinguish relevant paths from irrelevant ones and only reasons about the

Read the complete text