VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean
Yutong Xin, Qiaochu Chen, Greg Durrett, Işil Dillig · 2026
Read VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean by Yutong Xin, Qiaochu Chen, Greg Durrett, Işil Dillig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-18307.txt.
Description
Yutong Xin and three co-authors built VeriSoftBench, a benchmark for proving software correct in Lean. Their 2026 paper notes that language models shine at interactive theorem proving, but most benchmarks draw on mathematics from Mathlib, while software proofs live inside definition-rich codebases with large project-specific libraries. The benchmark gathers five hundred Lean 4 proof tasks from open-source formal-methods projects, keeping realistic repository context and cross-file links.
Tests of frontier models and specialized provers yield three lessons. Mathlib-tuned provers transfer poorly to this repository-centered setting. Success tracks transitive repository dependence: proofs drawing on large, multi-hop dependency closures fail more often. Curated context limited to a proof's dependency closure beats showing the whole repository, yet leaves much room to grow. Benchmark and evaluation suite are public.
Catalog record
| CREATOR | Yutong Xin, Qiaochu Chen, Greg Durrett, Işil Dillig |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-18307