Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis

Ke Zhang, Patricio Gallardo, Maziar Raissi, Sudhir Murthy · 2026

Read Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis by Ke Zhang, Patricio Gallardo, Maziar Raissi, Sudhir Murthy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16538.txt.

Introduction

Zhang and colleagues study agents that translate plain mathematics into faithful Lean 4 code. The gap between set-theory instinct and strict type theory leads models to invent fake library names. They test three tool types: expert-draft querying, symbol-name search, and build feedback through a Lean REPL. The agent beats one-shot rivals by large margins on build success and semantic match. That split then shows each tool type's extra gain to overall scores.

Catalog record

CREATORKe Zhang, Patricio Gallardo, Maziar Raissi, Sudhir Murthy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43906
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16538

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Tool-Augmented Agents for Lean Formalization: A Factorial Analysis Ke Zhang, Patricio Gallardo, Maziar Raissi, Sudhir Murthy Abstract Automatic translation of natural language mathematics into faithful Lean 4 code is hindered by the fundamental dissonance between informal set-theoretic intuition and strict formal type theory. This gap often causes LLMs to hallucinate non-existent library definitions, resulting in code that fails to compile or lacks semantic fidelity. In this work, we investigate the effectiveness of tool-augmented agents for this task through a systematic factorial analysis of three distinct tool categories: Fine-tuned Model Querying (accessing expert drafts), Knowledge Search (retrieving symbol definitions), and Compiler Feedback (verifying code via a Lean REPL). We first benchmark the agent against one-shot baselines, demonstrating large gains in both compilation success and semantic equivalence. We then use the factorial decomposition to quantify the impact of each category, isolating the marginal contribution of each tool type to overall performance. 1 Introduction Large language models (LLMs) have demonstrated remarkable capability in producing complex mathematical arguments ( Castelvecchi 2025 ) , yet their probabilistic nature fundamentally conflicts with the absolute certainty required by formal mathematics. Proof assistants such as Lean 4 ( de Moura & Ullrich 2021 ) resolve this conflict by mechanically verifying logical correctness, but they introduce a significant barrier: the ”formalization bottleneck.” Most modern undergraduate and graduate-level theorems exist only in natural language and have not yet been translated into the strict syntax required by proof assistants ( Lean Prover Community ) . Current attempts to automate translation into Lean 4 rely primarily on one-shot prompting or fine-tuning on static datasets ( Gao et al. 2025 ; Wang et al. 2025 ) . Although these methods can generate plausible code, they face several fundamental obstacles. Training data for graduate-level mathematics is sparse, causing models to hallucinate nonexistent theorems and symbols. Even when the mathematics is correct, a deep mismatch remains between the set-theoretic language of textbooks and Lean’s dependent type theory, making direct translation unreliable. Moreover, syntactic validity does not ensure semantic fidelity: Lean code may compile yet express a vacuous or incorrect statement. These issues are further compounded by the fact that Mathlib 4 is a fast-evolving, community-maintained library, so models trained on static snapshots frequently generate code that targets deprecated or nonexistent definitions in the current environment. To address these limitations, we introduce an agentic framework that replaces static generation with iterative refinement. A generalist LLM orchestrator interacts with three tool categories—symbol retrieval, expert drafting, and Lean REPL feedback—forming a closed loop of draft, verify, and repair that corrects both syntactic and semantic errors. Rather than treating this toolset as a black box, we apply a full factorial design on a new benchmark of 400 graduate-level theorems to quantify the causal contribution and interaction of each component. This allows us to identify which tools create new capability regimes and which merely stabilize or accelerate convergence toward faithful formalizations. Finally, to support reproducibility, we will open-source our entire research framework, including the full benchmark, agent implementation, evaluation scripts, and agent tool calling logs upon acceptance. 2 Problem Setting We address the task of formalization : automatically translating natural-language mathematical statements into valid Lean 4 theorem declarations. This task serves as the critical first step in verifying mathematics, bridging the gap between informal human intent and machine-checkable syntax. 2.1 Task Definition Formally, let \mathcal{X} be the space of informal mathematical statements and \mathcal{Y} be the space of valid Lean 4 source code. Given an input x\in\mathcal{X} , the system must generate a formal statement y\in\mathcal{Y} that satisfies three criteria: it must possess syntactic validity (compile in Lean 4 with Mathlib), maintain statement-centricity (omitting proofs via := by sorry ), and ensure semantic faithfulness (logical equivalence to x ). 2.2 Dataset Construction To benchmark this task, we curated a dataset of 400 graduate-level theorems derived from open-source LaTeX lecture notes and textbooks. We selected sources that provide natural language statements without accompanying formal code, ensuring the task represents a genuine translation effort rather than retrieval. As detailed in Table 1 , we balanced the dataset across four major mathematical domains—Real Analysis, Complex Analysis, Topology, and Algebra—selecting 100 diverse examples from each to ensure broad coverage of mathematical terminology and structures. Table 1: Benchmark Dataset Composition. Domain Source Material N Real Analysis Basic Analysis (Lebl) ( Lebl 2025a ) 100 Complex Analysis Cultivating Complex Analysis (Lebl) ( Lebl 2025b ) 100 Topology Notes on Topology (McKay) ( McKay 2025 ) 100 Algebra Abstract Algebra (Doty) ( Doty 2025 ) 100 Total 400 2.3 Evaluation Protocol To evaluate the quality of the generated formalizations, we adopt a rigorous two-stage protocol. First, we apply a Compiler Verification filter; any code that fails to compile is immediately assigned a score of 0. Second, for compiling code, we employ an LLM-as-a-Judge (GPT-5.2 Medium Thinking mode) to assess semantic equivalence. The evaluator compares the generated Lean statement against the original natural language input and assigns a faithfulness score on a scale of 0–10. Because all reported results depend on this judge, we explicitly validate its reliability via cross-judge agreement with an independent model in (Section 4.5 ). The judge prompt and rubric (Appendix B ) were reviewed and adjusted by a Lean 4–experienced researcher to align the evaluation with statement-level meaning. Success Metric: We define a generated translation y for an input statement x as Faithful if and only if: \text{Compiles}(y)=\text{True}\quad\textbf{AND}\quad\text{EquivalentScore}(x,y)\geq 9 3 Methodology Figure 1 : Agent Orchestration Logic. The architecture consists of a central LLM orchestrator (left) and a Lean 4 execution environment (right). The agent architecture, illustrated in Figure 1 , is structured as a for-loop. The core component is a central LLM orchestrator (GPT 5.2) that interacts with the Lean 4 environment via a defined API. Unlike static prompting, this architecture allows the model to maintain a persistent state, observing the consequences of its actions—such as compiler errors or retrieval results—before deciding on the next step. This design effectively decouples the LLM reasoning from the Lean Compiler verification. 3.1 Tool Definitions We equip the agent with tools designed to bridge the gap between informal mathematical reasoning and formal Lean 4 syntax: • Mathlib Retrieval: Retrieval tools such as lean_inspect_name allow the agent to ask Lean “what is this symbol?”, returning its real type and definition directly from Mathlib. • Expert Drafting: The lean4_translator tool requests a translation from the fine-tuned model Herald, providing a specialized starting point. • Compiler Feedback: The lean_repl_runner tool exposes the Lean 4 compiler interface. This provides precise error messages, enabling the agent to iteratively diagnose and correct syntax or validity issues. Detailed specifications of the tool definitions and signatures are provided in Appendix A . 3.2 Prompt Composition To ensure rigorous evaluation, we employ a modular prompting architecture. The system context is dynamically assembled from three independent components, ensuring that the core task definition remains constant while we vary the available tools: • Role Definition: Instructs the agent to act strictly as a translator, not a prover. The objective is to produce a type-correct theorem statement ending in := by sorry , ignoring proof generation. • General Guidelines: Enforces validity standards, such as requiring import Mathlib and strictly prohibiting the invention of new definitions or axioms

Read the complete text