Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game
Lixing Li · 2026
Read Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game by Lixing Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00677.txt.
Description
Lixing Li asks whether language models that prove theorems in Lean are truly reasoning, or pattern-matching against memorized training material. To find out, Li built the Obfuscated Natural Number Game: the classic Natural Number Game, a gentle introduction to formal proof, with every identifier renamed. The renamed world is a closed, zero-knowledge environment where semantic cues from pretraining are stripped away, leaving only local axioms and definitions to reason from.
Li calls the skill under test architectural reasoning: synthesizing proofs purely from a domain's own rules. Every model tested paid a latency tax under obfuscation, taking longer to think. Robustness then split the field. General chat models lost accuracy without their semantic crutches. Reasoning-oriented models, including dedicated provers, held their accuracy despite the missing cues. The benchmark offers a quantitative way to tell genuine reasoning from recall, which matters for automated theorem discovery.
Catalog record
| CREATOR | Lixing Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.00677