RagVerus : Repository-Level Program Verification with LLMs using Retrieval Augmented Generation
Si Cheng Zhong, Jiading Zhu † \dagger, Yifang Tian † \dagger, Xujie Si · 2025
Read RagVerus : Repository-Level Program Verification with LLMs using Retrieval Augmented Generation by Si Cheng Zhong, Jiading Zhu † \dagger, Yifang Tian † \dagger, Xujie Si (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05344.txt.
Introduction
Sicheng Zhong and co-authors present a 2025 computing paper on proving programs correct with help from language models. Formal verification checks that code meets its spec, but whole projects span many modules with tangled links. Past tools focus on single functions and miss that wider context.
Their framework, RagVerus, pairs retrieval-augmented generation with context-aware prompting. When a proof is needed, the system pulls in relevant material from across the repository to guide the model. The team built a new benchmark of 383 proof tasks drawn from multi-module projects in the Verus language. On that set, the framework improved results by 27 percent over the base approach. It also tripled proof pass rates on older benchmarks within a tight budget.
Catalog record
| CREATOR | Si Cheng Zhong, Jiading Zhu † \dagger, Yifang Tian † \dagger, Xujie Si |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05344
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RagVerus : Repository-Level Program Verification with LLMs using Retrieval Augmented Generation Si Cheng Zhong, Jiading Zhu † \dagger, Yifang Tian † \dagger, Xujie Si Abstract Scaling automated formal verification to real-world projects requires resolving cross-module dependencies and global contexts, which are challenges overlooked by existing function-centric methods. We introduce RagVerus , a framework that synergizes retrieval-augmented generation with context-aware prompting to automate proof synthesis for multi-module repositories, achieving a 27% relative improvement on our novel RepoVBench benchmark—the first repository-level dataset for Verus with 383 proof completion tasks. RagVerus triples proof pass rates on existing benchmarks under constrained language model budgets, demonstrating a scalable and sample-efficient verification. 1 Introduction Good engineers favor well-written tests to confirm that code works correctly, at least for the tested inputs. In software verification, testing is replaced by a set of complete specification; a verifier conducts a compile-time check to ensure the implementation matches the specification, providing vital assurances especially for high-confidence systems in critical areas such as operating systems and financial applications [ 17 , 32 ] . However, proof construction is a challenging process that requires domain expertise [ 29 , 11 , 15 ] . This process is time-consuming and non-trivial especially for large software projects. Advances in generative AI for code completion have spurred interest in verification-aware program synthesis, aiming to simultaneously enhance code trustworthiness and automate proof generation [ 8 , 30 , 26 ] . Large language models (LLMs) streamline formal verification by rapidly sampling and iterating proof candidates, lowering barriers to adoption [ 4 , 1 , 3 ] . However, current function-level program verification approaches inherit vanilla retrieval models [ 32 , 3 ] from theorem proving that solely rely on finetuned embedding encoders using prior examples [ 31 ] , lacking the rich, precise domain knowledge of code syntax or project structure [ 24 ] . Moreover, existing datasets [ 21 ] are derived for small-scale, isolated contexts, lacking evaluation in complex repository settings to reflect real practices in production. We recognize the main challenges of repository-level verification tasks to be (1) the vast semantic context and (2) the identification of correct dependent premises, as noted in prior works studying large-scale verification in the theorem proving domain [ 17 , 31 , 10 ] . A major challenge in LLM proof generation is premise retrieval—selecting the minimal lemma set from a repository-level codebase that aids verification while fitting within the context window. Scaling verification to repository level adds complexity due to inter-dependencies in large codebases, while the lack of benchmark data limits furthur evaluatiosn in the area. To assess LLM-based formal repository-level program verification, we establish three main contributions in this paper: We propose RagVerus , a new verification framework integrating retrieval-augmented generation (RAG) with context-aware prompting to guide LLMs in synthesizing verifiable code for interdependent, large-scale codebases. We provide a configurable RAG pipeline to enable systematic evaluation of diverse retrieval strategies (e.g., semantic search, dependency graphs) within a sandboxed Verus environment, informing cross-module dependencies and project-specific invariants. We also construct RepoVBench , the first repository-level program verification benchmark, where we collect recent award-winning Verus projects and extract programs with complex dependencies. Specifically, we build a dataset containing 2073 functions with dependencies over 52 modules, supporting evaluation on 383 proof completion tasks. We further evaluate RagVerus on two benchmarks: VerusBench from AutoVerus [ 30 ] , which is a function-level verification benchmark, and the new repository-level verification benchmark, RepoVBench . On VerusBench , RagVerus demonstrates a 200+% improvement in proof pass rates compared to the non-RAG baseline under constrained sampling budgets. On RepoVBench , our framework produces contextually coherent proofs to solve 5% more of the benchmark (27% relative increase), outperforming isolated function-level approaches. 2 Background Verus [ 12 ] is an SMT-based verification tool designed to formally verify Rust programs while leveraging Rust’s powerful type system, in particular its linearity and borrow checking mechanisms. Users write its domain-specific language (DSL) inline with Rust, while dividing Rust code into three code modes: specification annotation, proof annotation, and executable implementation. Verification in Verus is performed by encoding Rust functions and their associated annotations as SMT formulas [ 5 ] , which are then processed by an SMT solver such as Z3 [ 6 ] . Recently, AutoVerus [ 30 ] , an LLM-based approach shows strong performance in automating proof completion in Verus solving nearly all tasks of VerusBench. RAG [ 14 ] is a novel technique to enhance generative models by incorporating external knowledge retrieval, enabling more accurate and contextually relevant responses in knowledge-intensive tasks. Existing RAG methods [ 21 , 32 ] used in the software verification domain are relatively simple, such as directly comparing the method signatures and post-conditions. We draw inspiration from repository-level code generation [ 24 ] and theorem proving [ 31 ] domains, examining a framework to capture the structured nature of programming tasks. 3 RagVerus Framework Figure 1 : Illustration of the RagVerus Framework Pipeline We propose RagVerus , a retrieval-augmented framework for LLM-based program verification, with a special focus on complex Verus projects. The framework modularizes verification task preparation, context retrieval methods, and proof generation models, leveraging repository information and verifier feedback to align domain knowledge. 3.1 The General Pipeline As illustrated in Fig. 1 , the RagVerus pipeline consists of three stages: 1) mining code properties, 2) retrieving task-specific information, and 3) finally generating proofs and/or verifiable code. Mining Code Properties . We preprocess repository-wide code artifacts to extract verification-critical metadata (more details in Section 4.1), building on insights from context modeling in repository-level code generation [ 24 ] . Through static analysis over the codebase hierarchy, we identifying function/type signatures, method calls, module dependencies, and control/dataflow relationships essential for proof construction. To enable efficient retrieval, RagVerus uses persistent vector stores where documents are encoded into embeddings using OpenAI’s text-embedding-3-large [ 23 ] , indexed via FAISS [ 7 ] . Depending on the retrieval method chosen, either the Rust code themselves or their metadata is indexed. This hybrid representation preserves both syntactic and semantic patterns in the form of high-dimensional embeddings, enabling adaptive context retrieval in later stages. Context Retrieval . We believe two general kinds of retrieval tasks are essential for repository verification. Few-shot example retrieval matches code snippets or metadata to provide contextual proof patterns and exemplar code-proof pairs, while Dependency retrieval identifies function signatures and domain-specific syntax necessary for constructing proofs. Retrieved contexts are integrated into prompts to guide proof generation, with examples expanded in Section 3.2 . Proof Generation . This stage accepts a generic code generation agent like AutoVerus . The framework processes Rust modules containing executable code and Verus specifications (pre-/post-conditions) without existing proof annotations. Language models synthesize verification annotations—loop invariants, assertions, and proof blocks—conditioned on retrieved contexts to produce a fully annotated Rust program. Verus compiler validates if specifications hold across all situations, with failed proofs triggering iterative refinement—feeding back compiler errors to the agent—if requested. While being evaluated on proof generation tasks, RagVerus can also support flexible extensions for code generation as well as specification inference. 3.2 Instantiations of Retrieval Module Building upon the modular framework established in Section 3.1, this subsection presents customizable retrieval modules tailored to address distinct