LeanCat: A Benchmark Suite for Formal Category Theory in Lean (Part I: 1-Categories)

Rongge Xu, Hui Dai, Yiming Fu, Jiedong Jiang, Tianjiao Nie, Hongwei Wang, ​ Junkai Wang, ​ Holiverse Yang, ​ Jiatong Yang, ​ Zhi-Hao Zhang · 2025

Read LeanCat: A Benchmark Suite for Formal Category Theory in Lean (Part I: 1-Categories) by Rongge Xu, Hui Dai, Yiming Fu, Jiedong Jiang, Tianjiao Nie, Hongwei Wang, ​ Junkai Wang, ​ Holiverse Yang, ​ Jiatong Yang, ​ Zhi-Hao Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24796.txt.

Introduction

Rongge Xu, Hui Dai, Yiming Fu, and co-authors introduce LeanCat, a benchmark of one hundred formal category theory tasks written in the Lean proof assistant. Category theory tests library-grounded abstraction, meaning reasoning with high-level interfaces and reusable structures. Existing theorem proving benchmarks rarely measure that skill, favoring algebra or arithmetic instead.

Evaluation exposes a severe abstraction gap: the best model solved only a small share of tasks, and success fell sharply on harder structural problems. Readers in AI and formal methods get a stress test for structural reasoning. The suite is part one, covering ordinary categories, with more parts implied.

Catalog record

CREATORRongge Xu, Hui Dai, Yiming Fu, Jiedong Jiang, Tianjiao Nie, Hongwei Wang, ​ Junkai Wang, ​ Holiverse Yang, ​ Jiatong Yang, ​ Zhi-Hao Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-24796

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LeanCat: A Benchmark Suite for Formal Category Theory in Lean (Part I: 1-Categories) Rongge Xu, Hui Dai, Yiming Fu, Jiedong Jiang, Tianjiao Nie, Hongwei Wang, ​ Junkai Wang, ​ Holiverse Yang, ​ Jiatong Yang, ​ Zhi-Hao Zhang Abstract Large language models (LLMs) have made rapid progress in formal theorem proving, yet current benchmarks under-measure the kind of abstraction and library-mediated reasoning that organizes modern mathematics. In parallel with FATE’s emphasis on frontier algebra, we introduce LeanCat 1 1 1 LeanCat (Part I: 1-Categories) is open sourced at https://github.com/sciencraft/LeanCat . , a Lean benchmark for category-theoretic formalization—a unifying language for mathematical structure and a core layer of modern proof engineering—serving as a stress test of structural, interface-level reasoning. Part I: 1-Categories contains 100 fully formalized statement-level tasks, curated into topic families and three difficulty tiers via an LLM-assisted + human grading process. The best model solves 8.25 % 8.25\% of tasks at pass@1 ( 32.50 % / 4.17 % / 0.00 % 32.50\%/4.17\%/0.00\% by Easy/Medium/High) and 12.00 % 12.00\% at pass@4 ( 50.00 % / 4.76 % / 0.00 % 50.00\%/4.76\%/0.00\% ). We also evaluate LeanBridge which use LeanExplore to search Mathlib, and observe consistent gains over single-model baselines. LeanCat is intended as a compact, reusable checkpoint for tracking both AI and human progress toward reliable, research-level formalization in Lean. 1 Introduction Recent advances in large language models (LLMs) and agentic training have revived the prospect of end-to-end formal theorem proving. On the formal side, systems such as OpenAI’s early Lean-based prover ( 17 ) and DeepMind’s AlphaProof ( 8 ) demonstrate that reinforcement learning with formal verification feedback can produce non-trivial Lean proofs. More recently, specialized provers such as Seed-Prover 1.5 ( 5 ) show that large-scale agentic RL and test-time scaling can push formal success rates on established benchmarks substantially higher. These results suggest that formal proof generation is no longer limited to toy domains, and that tight tool feedback loops (retrieve–generate–verify) can be a decisive ingredient. Despite steady progress in neural theorem proving, current formal benchmarks still do not adequately exercise library-grounded, abstraction-heavy reasoning. Widely used datasets such as miniF2F ( 22 ) and FIMO 12 largely draw from olympiad-style problems, while university-level suites like ProofNet 4 and PutnamBench 20 focus on undergraduate competition or textbook material. These benchmarks are valuable, but they often reward short clever tricks or computation rather than sustained work inside rich abstract frameworks. By contrast, modern research mathematics is written at high generality, organized around reusable interfaces, and deeply intertwined with large libraries of definitions and lemmas—so success hinges less on a single key insight and more on navigating abstraction, managing definitions, and coherently composing library knowledge over long horizons. Category theory provides a natural stress-test for this capability: as the interface language of modern mathematics—categories, functors, natural transformations, adjunctions, limits/colimits, monads—formal proofs typically rely on diagrammatic and universal-property reasoning, i.e., constructing morphisms with the right naturality/uniqueness guarantees and showing commutativity across families of structures. Yet existing formal benchmarks rarely target this abstraction layer directly. To bridge this gap, we introduce LeanCat , a benchmark of 100 category-theory problems formalized in Lean 4 (mathlib), designed to test whether automated provers can operate inside a mature library and compose high-level abstractions rather than solve isolated puzzles. LeanCat complements algebra-focused benchmarks such as the FATE series 9 by shifting the frontier from abstract algebra to category theory. Our baseline evaluation reveals a stark abstraction gap: across five strong models, the best achieves only 8.25\% at pass@1 and 12\% at pass@4, with accuracy dropping sharply from Easy to High once library navigation and long-horizon abstraction management become necessary (Figure 1 ). We further observe a persistent gap between producing a plausible natural-language argument and generating compilable Lean code, highlighting a pronounced natural-to-formal bottleneck (Figure 2 ). Figure 1: Formal proof accuracy (pass@4) across LeanCat model baselines. Figure 2: Intermediate natural-language (NL) vs. formal-language (FL) (pass@4) accuracy on LeanCat, highlighting the natural-to-formal gap. To our knowledge, LeanCat is the first installment of a category-theory benchmark series . This paper focuses on on 1-category theory. We envision future extensions targeting richer structures such as monoidal categories, enriched and braided settings, and eventually 2-categories and higher-categorical interfaces. Beyond benchmarking, we view this direction as meaningful for both (i) human mathematics , by mapping what parts of abstract library-level reasoning remain hard to formalize and where libraries need strengthening, and (ii) AI , by forcing progress on abstraction-aware planning, retrieval of relevant lemmas, and robust compilation-driven refinement. Our key contributions are summarized as follows: • LeanCat benchmark (1-Categories). We present LeanCat, a set of 100 category-theory problems formalized in Lean 4 (mathlib 4.19.0). The tasks span eight topic clusters (from basic categorical properties to monads), curated to cover reusable abstract interfaces rather than contest-style tricks. • Difficulty annotation pipeline. We propose a grading process combining model-based estimates and expert judgment. Each problem receives multiple 1–10 ratings (from advanced LLM attempts and from human formalizers), aggregated with heavier weight on human scores to assign Easy/Medium/High labels (20/42/38). • Baseline evaluation. We benchmark state-of-the-art provers on LeanCat under uniform conditions. Our evaluation (Section 3-4) includes ChatGPT-5.1 and ChatGPT-5.2 ( 15 ; 16 ) , Claude 4.5 ( 3 ) , Google’s Gemini 3 pro ( 7 ) , an advanced chain-of-thought reasoner DeepSeek -V3.2-Thinking and DeepSeek V3.2 Speciale ( 11 ) and ann agentic model Kimi K2 19 . At pass@1, the best model solves 8.25% of LeanCat; at pass@4, the best reaches 12% . We provide breakdowns by difficulty and identify dominant failure modes (library gaps, abstraction mismatch, and stalled multi-step plans). • Search-augmented proving via LeanBridge. We evaluate a retrieve–analyze–generate–verify loop that integrates mathlib retrieval (via LeanExplore) and compiler feedback, illustrating how tool-augmented workflows can improve robustness on a subset of problems. 2 LeanCat Benchmark Design 2.1 Benchmark Structure and Content Data Source The problems in our benchmark are organized into two main parts: Abstract and Concrete: • Abstract part : Problems are primarily drawn from standard, widely used textbooks in category theory, especially Category Theory in Context ( 18 ) and Categories for the Working Mathematician ( 13 ) , together with a small number of problems adapted from unpublished lecture notes ( 10 ; 21 ) . • Concrete part : Problems are curated mainly from Abstract and Concrete Categories ( 2 ) , which provides a systematic source of exercises on concretizability, injectivity, and related themes. • Others : Beyond these core sources, we also include selected problems inspired by research papers and advanced community-driven references ( 6 ; 1 ) . Each LeanCat problem is self-contained at the statement level: the formal statement of the theorem is provided (often with informal description as in the Problem listing above), and all necessary definitions exist in Lean’s environment (either already in Mathlib ( 14 ) or introduced as part of the problem setup). Where possible, we drew on known theorems from category theory literature, many tasks were crafted or adapted specifically to probe corner cases and interface interactions that an AI prover might struggle with. In several High-difficulty cases, relevant lemmas were not readily available, forcing human formalizers to devise intermediate results. This aspect of LeanCat, that it

Read the complete text