HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement

Jilin Hu, Jianyu Zhang, Yongwang Zhao, Talia Ringer · 2025

Read HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement by Jilin Hu, Jianyu Zhang, Yongwang Zhao, Talia Ringer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15740.txt.

Introduction

Jilin Hu and co-authors present a 2025 paper on a tool that helps computers write formal math proofs. Proof writing by hand is slow, so such tools matter for checking key systems. Past tools either built proofs step by step or wrote a whole proof at once, and each style has gaps.

Their system, HybridProver, joins both styles through a proof sketch that sits between them. Parts of a sketch that are right can be kept and refined. The team trained two mid-size language models on cleaned proof data. On a standard test set the system solved far more tasks than the prior best. Tests also show that data quality and sampling choices drive much of the gain.

Catalog record

CREATORJilin Hu, Jianyu Zhang, Yongwang Zhao, Talia Ringer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15740

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HybridProver: Augmenting Theorem Proving with LLM-Driven Proof Synthesis and Refinement Jilin Hu, Jianyu Zhang, Yongwang Zhao, Talia Ringer Abstract Formal methods is pivotal for verifying the reliability of critical systems through rigorous mathematical proofs. However, its adoption is hindered by labor-intensive manual proofs and the expertise required to use theorem provers. Recent advancements in large language models (LLMs) offer new opportunities for automated theorem proving. Two promising approaches are generating tactics step by step and generating a whole proof directly with an LLM. However, existing work makes no attempt to combine the two approaches. In this work, we introduce HybridProver , a dual-model proof synthesis framework that combines tactic-based generation and whole-proof synthesis to harness the benefits of both approaches. HybridProver generates whole proof candidates for evaluation directly, then extracts proof sketches from those candidates. It then uses a tactic-based generation model that integrates automated tools to complete the sketches via stepwise refinement. We implement HybridProver for the Isabelle theorem prover and fine-tune LLMs on our optimized Isabelle datasets. Evaluation on the miniF2F dataset illustrates HybridProver ’s effectiveness. We achieve a 59.4% success rate on miniF2F, where the previous SOTA is 56.1%. Our ablation studies show that this SOTA result is attributable to combining whole-proof and tactic-based generation. Additionally, we show how the dataset quality, training parameters, and sampling diversity affect the final result during automated theorem proving with LLMs. All of our code, datasets, and LLMs are open source. uto counter, list inside=mycode]mylisting[2][] boxsep=0.5mm, left=1mm, top=-1.5mm, bottom=-2mm, colback=gray!10, colframe=black!60, arc=3pt, boxrule=1.5pt, enhanced, drop fuzzy shadow, title= Listing \thetcbcounter : #2, label=listing: \thetcbcounter , list entry= Listing \thetcbcounter #2, listing only, listing options= basicstyle= , mathescape=true, escapeinside=%**), columns=fullflexible, numbers=none, xleftmargin=0em, keywordstyle= , keepspaces=true, breaklines=true, breakautoindent=false, breakindent=0pt, literate=* __1 , #1 1 Introduction The field of formal methods plays a pivotal role in ensuring the reliability of critical systems across mathematics, computer science, and engineering Bjørner and Havelund 2014 . It rigorously verifies system correctness through formal specification and math-based proof. However, the labor-intensive nature of manual proof construction and the expertise required for the effective use of interactive theorem provers (ITPs) present significant barriers to widespread adoption. To address this challenge, automated theorem proving has emerged as a paradigm to streamline formal verification Barrett et al. 2011 ; De Moura and Bjørner 2008 ; Groom 2018 ; Schulz 2002 ; Weidenbach et al. 1996 , but its large proof-search space and limited adaptability to diverse proof scenarios have constrained its practicality. Recent advances in large language models (LLMs) have opened new frontiers for automated theorem proving. Generating proofs is similar to code generation, where LLMs have shown outstanding capabilities Chen et al. 2021 ; Jiang et al. 2024 . Moreover, formal proofs are written in formal languages that can be verified by ITPs, e.g., Isabelle/HOL Nipkow et al. 2002 , Rocq Sozeau et al. 2025 , Lean Moura and Ullrich 2021 , and HOL Light Harrison 1996 . Thus, the dangers of LLM hallucinations can be significantly reduced in this process. In this work, we introduce HybridProver , an LLM-driven automated theorem-proving framework. Before HybridProver , two predominant approaches had emerged for LLM-driven automated theorem-proving: (1) whole-proof synthesis that produces complete formal proof attempts Xin et al. 2024 ; First et al. 2023 ; Wang et al. 2024b , and (2) tactic-based generation that guides LLMs stepwise Yang et al. 2023 ; Zhao et al. 2024b ; Jiang et al. 2023 ; Polu and Sutskever 2020 . Whole-proof synthesis excels at high-level planning, but is error-prone at the level of details; tactic-based generation excels at the details, but may struggle with higher-level planning. HybridProver is the first system that we are aware of to combine these two approaches. In doing so, HybridProver gets the best of both worlds—with state-of-the-art (SOTA) results to show for it. HybridProver achieves this by employing a dual-model architecture: First, HybridProver uses a whole-proof synthesis model to generate candidate proofs based on the user-provided theorem statement. If any of these candidates are correct whole proofs, then HybridProver is done. Otherwise, HybridProver mechanically abstracts details from those candidate proofs to build proof sketches—abstract proof plans with subgoals in which the detailed tactics are omitted. HybridProver uses the ITP to check the proof sketch to ensure its correctness. Finally, HybridProver uses a tactic-based generation model to refine the proof sketch with tactic details and produce a correct whole proof, checked by the ITP. In this way, HybridProver uses the notion of a proof sketch to bridge the gap between the two approaches. As a consequence, the core ideas of HybridProver can be used to compose any whole-proof synthesis model with any tactic-based generation model for the same target ITP. This is in contrast with prior work generating proof sketches and filling in the details Jiang et al. 2022 ; Jiang et al. 2023 ; Mikuła et al. 2023 ; Yang et al. 2023 , which does not offer a way to repurpose existing whole-proof synthesis models to generate proof sketches. Nonetheless, HybridProver can also take advantage of the benefits of these prior works by integrating hammers Blanchette et al. 2016 , existing tools that automate the proving of simple theorems using symbolic techniques. While the ideas for HybridProver can be implemented for many ITPs, we focus on Isabelle/HOL Nipkow et al. 2002 , using Sledgehammer Paulson 2012 as the hammer. We choose Isabelle both because of the unique challenges that it presents for LLM-based proof synthesis, as well as the unique advantages that it provides for curating an optimized dataset (see Section 2 ). We evaluate HybridProver on miniF2F Zheng et al. 2022 , a popular mathematical theorem proving dataset. We improve the SOTA on miniF2F from 56.1% to 59.4%. We show that the combination of whole-proof synthesis and tactic-based generation improves the proof performance. With the integration of hammers, HybridProver performs even better. Additional ablation studies show how the dataset quality, training parameters, and sampling diversity affect the final proof result. We open-source our optimized Isabelle datasets and fine-tuned LLMs on Huggingface. We also provide the source code for HybridProver . Contributions: (1) We propose HybridProver , an LLM-driven automated theorem proving framework that combines whole-proof synthesis and tactic-based generation, using proof sketches as a bridge between the two approaches to get the best of both worlds. (2) We implement HybridProver for Isabelle/HOL, fine-tuning LLMs on our custom-optimized Isabelle datasets. Our evaluation shows that HybridProver brings the SOTA results on miniF2F for Isabelle up from 56.1% to 59.4%. (3) Our ablation studies show the effectiveness of HybridProver and how the data quality, training parameters, and sampling diversity affect LLM’s ability in theorem proving. 2 Background: why Isabelle? We implement HybridProver to automate formal proof in the Isabelle ITP. ITPs like Isabelle Nipkow et al. 2002 , Rocq Sozeau et al. 2025 , and Lean Moura and Ullrich 2021 are specialized systems that make it possible to formally prove software systems and mathematical theorems. They rigorously check these formal proofs using small, trustworthy logic checkers Nipkow et al. 2002 ; Pierce et al. 2010 . Unfortunately, using these ITPs requires substantial work Klein et al. 2009 ; Gu et al. 2016 ; Leroy et al. 2016 . Complementary

Read the complete text