Automating Mathematical Proof Generation Using Large Language Model Agents and Knowledge Graphs

Vincent Li, Yule Fu, Tim Knappe, Kevin Han, Kevin Zhu · 2025

Read Automating Mathematical Proof Generation Using Large Language Model Agents and Knowledge Graphs by Vincent Li, Yule Fu, Tim Knappe, Kevin Han, Kevin Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11657.txt.

Introduction

Vincent Li and co-authors give language models a knowledge graph to help them write math proofs, in this 2025 paper. Proof writing demands spotting key ideas and how they connect, which models often fumble. Their framework, KG-Prover, mines graphs of concepts from trusted math texts and feeds them to a general model.

The study also scales up the graph-based thinking time the model gets at test time. Across three data sets, adding the graphs lifted scores, with one test improving by up to twenty-one percent. Paired with a strong small model, the system solved half the problems in one benchmark. No extra fine-tuning was needed.

Catalog record

CREATORVincent Li, Yule Fu, Tim Knappe, Kevin Han, Kevin Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11657

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Automating Mathematical Proof Generation Using Large Language Model Agents and Knowledge Graphs Vincent Li, Yule Fu, Tim Knappe, Kevin Han, Kevin Zhu Abstract Large Language Models have demonstrated remarkable capabilities in natural language processing tasks, including mathematical problem-solving that requires multi-step logical reasoning. However, challenges persist in automating the identification of key mathematical concepts, understanding their interrelations, and formalizing proofs within a rigorous framework. We present a novel framework that leverages knowledge graphs to augment LLMs to construct and formalize mathematical proofs. Our results demonstrate significant performance improvements across multiple datasets, with using knowledge graphs, achieving up to a 34% success rate on the MUSTARDSAUCE dataset on o1-mini and consistently outperforming baseline approaches by 2-11% across different models. We show how this approach bridges the gap between natural language understanding and formal logic proof systems and achieve elevated results for foundation models over baseline. 1 Introduction The advent of Large Language Models has revolutionized natural language processing, enabling machines to perform complex reasoning tasks ( Peters et al. 2018 ; Brown et al. 2020 ; Srivastava et al. 2023 ) . These models have shown promise in mathematical problem-solving, which inherently requires multi-step logical inference and precise understanding of abstract concepts. Despite these advancements, significant challenges remain in automating the identification of mathematical concepts, understanding their interrelations, and formalizing proofs within a mathematical framework ( Hendrycks et al. 2021 ) . Recent advancements in AI-driven mathematics focus on improving reasoning capabilities in language models and integrating them with formal verification systems. Novel approaches leverage reinforcement learning to address theorem proving ( Bansal et al. 2019 ) . Others enhance the understanding of mathematical concepts by combining semantic parsing with knowledge representation ( Johnson et al. 2020 ) . Training language models to generate formal proofs has underscored the difficulties of handling complex reasoning tasks without structured guidance ( Polu and Sutskever 2020 ) . Figure 1: Overview of the proof generation and formalization workflow. Nonetheless, existing methodologies often lack a comprehensive approach to extracting and structuring mathematical content from extensive knowledge bases, such as ProofWiki, and leveraging this structured data for proof generation ( ProofWiki 2025 ) . In this paper, we propose a framework that automates mathematical proof construction by combining Large Language Models with knowledge graphs. Our approach parses constructs a knowledge graph that captures mathematical relationships. We then leverage this graph to formalize proofs, integrating with Lean ( de Moura and Ullrich 2021 ) a theorem prover and programming language designed for formal proof verification. More specifically we: • Built a knowledge graph of over 60,000 nodes and 300,000 edges that represents mathematical concepts and their interrelations, facilitating traversal to alike subjects. • Integrate a framework of multiple language models working together towards a correct formal proof, achieving a 7-10% absolute gain over baseline approaches. • Utilize feedback-like approaches granting additional traversals for failure correction, allowing our knowledge graph method to consistently outperform the baseline. Our work aims to bridge the gap between natural language understanding and formal logic. We provide a detailed methodology, evaluate the effectiveness of our framework, and discuss its scalability and potential impact on the field of automated theorem proving. 2 Related Work The intersection of artificial intelligence and mathematics has garnered significant attention, with various research efforts aiming to automate mathematical reasoning and proof generation. DeepMind introduced HOList for higher-order theorem proving. By integrating reinforcement learning with neural networks, the system learns to predict useful tactics for proving theorems in the HOL Light theorem prover ( Bansal et al. 2019 ; Harrison 2009 ) . Further advancements include DeepMath , which combines LLMs with reinforcement learning to navigate proof spaces ( Lample and Charton 2022 ) . The MUSTARD project ( Johnson et al. 2020 ) aims to improve AI’s understanding of mathematics through semantic representation. By first generating a problem, constructing an informal proof, converting it into Lean ( de Moura et al. 2015 ) format, and verifying the proof with a Lean interpreter. Recent work explored using generative language models for automated theorem proving, by training transformer models on formal mathematical languages, equipping models such as DeepSeek-Prover-V1.5 with Methods like proof-assistant feedback to improve performance ( Polu and Sutskever 2020 ) . While previous research has made significant strides in automating aspects of mathematical reasoning, challenges remain in integrating natural language mathematical knowledge with formal proof systems ( Yang et al. 2023 ) . 3 Methodology Our framework automates mathematical proof generation by integrating Large Language Models with a knowledge graph constructed from ProofWiki. We employ a multi-stage approach combining retrieval-augmented generation with a two-agent system for proof formalization. The system consists of three main components: search strategy implementation, proof generation, and proof formalization. Figure 1 illustrates the overall workflow. 3.1 Overall Workflow Our main workflow integrates the components as follows, given a mathematical problem input: 1. Context Retrieval : Semantic search retrieves relevant context from the knowledge graph. 2. Informal Proof Generation : The LLM generates an informal proof using the context. 3. Formal Proof Generation : The Autoformalizer converts the informal proof into a formal proof. 4. Verification : The formal proof is verified using a proof verifier (e.g Lean) 5. Iterative Refinement : If verification fails, we retrieve another node and the process is iterated to improve the proof. 3.2 Knowledge Graph 3.2.1 Retrieval To retrieve relevant context for a given problem, we implemented a semantic search strategy leveraging the embeddings, by generating an embedding vector \mathbf{v}_{q} for problem statements and comparing to the problem embedding \mathbf{v}_{q} and the embeddings \mathbf{v}_{i} of the nodes in the knowledge graph using the cosine similarity function : \displaystyle sim(v_{q},v_{i})=\frac{v_{q}\cdot v_{i}}{\|v_{q}\|\|v_{i}\|} We then select the top k nodes with the highest similarity scores, retrieving the most relevant concepts related to the problem. The content of these selected nodes is injected to the context that will be provided to the language model. To further enrich the context with foundational concepts, we introduce a depth parameter d , which controls the traversal of the graph by limiting or expanding the allowed steps between nodes. This enables the inclusion of proofs or relationships that are located further from the root node. 3.2.2 Graph database We parsed ProofWiki to extract mathematical definitions, theorems, proofs, and related content, focusing on name-spaces corresponding to definitions, axioms, and proofs 1 1 1 Our constructed dataset shape can be referred to in Appendix F.1 . We use Neo4j, as a graph database, to store and manage the nodes and relationships, forming our knowledge graph ( Webber 2012 ) . Nodes are created with their respective properties, and relationships are established based on internal links within the content, capturing the inter-dependencies among concepts. We store the nodes in Neo4j alongside with their embedding vectors, enabling queries based on semantic similarity. Relationships between nodes were established 2 2 2 An example entry from our nodes collection can be found in Appendix D . 3.3 Proof Generation with LLMs 3.3.1 Informal Proof Generation The Informal proof generation integrates retrieved context into the language model prompt and uses the LLM to create a proof based on this enhanced input. If the proof is unsuccessful or incomplete, the framework iteratively deepens the context by

Read the complete text