APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning
Azim Ospanov, Farzan Farnia, Roozbeh Yousefzadeh · 2025
Read APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning by Azim Ospanov, Farzan Farnia, Roozbeh Yousefzadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05758.txt.
Introduction
Azim Ospanov, Farzan Farnia, and Roozbeh Yousefzadeh present a proof-repair system in a 2025 paper. Writing proofs that a formal checker like Lean accepts is hard for language models; the usual fix is to sample thousands of attempts until one passes. Their system, APOLLO, instead fixes broken proofs step by step.
A team of agents reads the checker's complaints, repairs syntax, isolates the failing sub-goals, calls solvers, and asks the model to retry only the stuck parts on a small budget. Fixed pieces are rejoined and rechecked, looping a set number of times. On a standard benchmark, APOLLO set a new best score for small models while using fewer than a hundred samples.
Catalog record
| CREATOR | Azim Ospanov, Farzan Farnia, Roozbeh Yousefzadeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05758
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning Azim Ospanov, Farzan Farnia, Roozbeh Yousefzadeh Abstract Formal reasoning and automated theorem proving constitute a challenging subfield of machine learning, in which machines are tasked with proving mathematical theorems using formal languages like Lean. A formal verification system can check whether a formal proof is correct or not almost instantaneously, but generating a completely correct formal proof with large language models (LLMs) remains a formidable task. The usual approach in the literature is to prompt the LLM many times (up to several thousands) until one of the generated proofs passes the verification system. In this work, we present APOLLO ( A utomated P r O of repair via L LM and L ean c O llaboration), a modular, model - agnostic pipeline that combines the strengths of the Lean compiler with an LLM’s reasoning abilities to achieve better proof-generation results at a low sampling budget. Apollo directs a fully automated process in which the LLM generates proofs for theorems, a set of agents analyze the proofs, fix the syntax errors, identify the mistakes in the proofs using Lean, isolate failing sub - lemmas, utilize automated solvers, and invoke an LLM on each remaining goal with a low top - K K budget. The repaired sub - proofs are recombined and reverified, iterating up to a user - controlled maximum number of attempts. On the miniF2F benchmark, we establish a new state - of - the - art accuracy of 75.0% among 7B - parameter models while keeping the sampling budget below one thousand. Moreover, Apollo raises the state - of - the - art accuracy for Goedel - Prover - SFT to 65.6% while cutting sample complexity from 25,600 to a few hundred. General - purpose models (o3 - mini, o4 - mini) jump from 3–7% to over 40% accuracy. Our results demonstrate that targeted, compiler - guided repair of LLM outputs yields dramatic gains in both efficiency and correctness, suggesting a general paradigm for scalable automated theorem proving. 1 Introduction Figure 1: The summary of whole-proof generation pipeline vs. proposed Apollo agentic pipeline. LLM refers to a chosen formal theorem generator model. Formal reasoning has emerged as one of the most challenging fields of AI with recent achievements such as AlphaProof and AlphaGeometry doing well at the International Math Olympiad competing with humans [ 1 , 2 , 3 ] . Formal reasoning relies both on AI models and a formal verification system that can automatically verify whether a mathematical proof is correct or not. In recent years, formal verification systems such as Lean [ 4 ] have facilitated a new form of doing mathematical research by enabling mathematicians to interact with the formal verification system and also with each other via the system, enabling larger numbers of mathematicians to collaborate with each other on a single project. As such, these formal verification systems are also called proof assistants as one can use them interactively to write a formal proof and instantaneously observe the current state of the proof and any possible errors or shortcomings in the proof generated by the compiler. Immediate access to the output of Lean compiler can help a mathematician to fix possible errors in the proof. At the same time, when a proof passes the compiler with no error, other collaborators do not need to verify the correctness of the proof. This type of collaboration is transforming the field of mathematics enabling large groups of collaborators to engage in large projects of mathematical research such as the Prime Number Theorem And More project [ 5 ] . Moreover, it has given rise to the digitization of mathematics [ 6 ] . In the AI front, large language models (LLMs) are improving at mathematical reasoning in natural language, and at the same time, they have shown a remarkable ability to learn the language of those formal verification systems and write mathematical proofs in formal language. This has led to the field of Automated Theorem Proving (ATP) where the standard approach is to prompt the LLM to generate a number of candidate proofs for a given theorem which will then be verified automatically using the formal verification system. Better models, better training sets, and better training methods has led to significant advances in this field [ 7 , 8 , 9 , 10 , 11 ] . Despite these advances, the LLMs do not really get the chance to interact with the verification system the way a human does. LLMs generate many possible proofs, sometimes as many as tens of thousands, and the verification system only marks the proofs with a binary value of correct or incorrect. This common approach is illustrated in Figure 1 . When the formal verification system analyzes a proof, its compiler may generate a long list of issues including syntax errors, incorrect tactics, open goals, etc. In the current approach of the community, all of those information/feedback from the compiler remains unused, as the LLM’s proof generation is not directly engaged with the compiler errors of the formal verification system. In this work, we address this issue by introducing Apollo, a fully automated system which uses a modular, model-agnostic pipeline that combines the strengths of the Lean compiler with the reasoning abilities of LLMs. Apollo directs a fully automated process in which the LLM generates proofs for theorems, a set of agents analyze the proofs, fix the syntax errors, identify the mistakes in the proofs using Lean, isolate failing sub - lemmas, utilize automated solvers, and invoke an LLM on each remaining goal with a low top - K budget. The high-level overview of Apollo appears at the bottom of Figure 1 . Our contributions are as follows: • We introduce a novel fully automated system that directs a collaboration between LLMs, Lean compiler, and automated solvers for automated theorem proving. • We evaluate Apollo on miniF2F-test set using best LLMs for theorem proving and demonstrate that Apollo improves the baseline accuracies by significant margins while keeping the sampling costs much lower. • We establish a new SOTA on miniF2F-test benchmark for medium-sized language models. (with parameter size 7B or less) 2 Related Works Formal theorem proving systems. At their core, formal theorem - proving (FTP) systems employ interactive theorem provers (ITPs) to verify mathematical results. In particular, Lean4 [ 4 ] is both a functional programming language and an ITP. Each theorem, lemma, conjecture, or definition must be checked by Lean’s trusted kernel, so the compiler’s verdict is binary: either the statement type - checks (True) or it fails (False). This rigorous verification dramatically increases the reliability of formal mathematics; however, it also increases the complexity of proof development: authors must both comprehend the mathematical concepts and precisely encode them in the formal language. The usefulness of Lean is also shown to go beyond theorem proving, as Jiang et al. [ 12 ] showed that Lean can be adapted to natural language logical reasoning. Patching broken programs. Many prior works have explored using feedback to repair broken proofs. In software engineering, “repair” typically refers to program repair, i.e. fixing code that no longer runs [ 13 ] . A common