Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game

Lixing Li · 2026

Read Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game by Lixing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00677.txt.

Description

Lixing Li asks whether language models that prove theorems in Lean are truly reasoning, or pattern-matching against memorized training material. To find out, Li built the Obfuscated Natural Number Game: the classic Natural Number Game, a gentle introduction to formal proof, with every identifier renamed. The renamed world is a closed, zero-knowledge environment where semantic cues from pretraining are stripped away, leaving only local axioms and definitions to reason from.

Li calls the skill under test architectural reasoning: synthesizing proofs purely from a domain's own rules. Every model tested paid a latency tax under obfuscation, taking longer to think. Robustness then split the field. General chat models lost accuracy without their semantic crutches. Reasoning-oriented models, including dedicated provers, held their accuracy despite the missing cues. The benchmark offers a quantitative way to tell genuine reasoning from recall, which matters for automated theorem discovery.

Catalog record

CREATORLixing Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00677

Browse the full library · About the Tower of Babel