VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean

Yutong Xin, Qiaochu Chen, Greg Durrett, Işil Dillig · 2026

Read VeriSoftBench: Repository-Scale Formal Verification Benchmarks for Lean by Yutong Xin, Qiaochu Chen, Greg Durrett, Işil Dillig (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-18307.txt.

Description

Yutong Xin and three co-authors built VeriSoftBench, a benchmark for proving software correct in Lean. Their 2026 paper notes that language models shine at interactive theorem proving, but most benchmarks draw on mathematics from Mathlib, while software proofs live inside definition-rich codebases with large project-specific libraries. The benchmark gathers five hundred Lean 4 proof tasks from open-source formal-methods projects, keeping realistic repository context and cross-file links.

Tests of frontier models and specialized provers yield three lessons. Mathlib-tuned provers transfer poorly to this repository-centered setting. Success tracks transitive repository dependence: proofs drawing on large, multi-hop dependency closures fail more often. Curated context limited to a proof's dependency closure beats showing the whole repository, yet leaves much room to grow. Benchmark and evaluation suite are public.

Catalog record

CREATORYutong Xin, Qiaochu Chen, Greg Durrett, Işil Dillig
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-18307

Browse the full library · About the Tower of Babel