Reasoning Structure of Large Language Models
Frédéric Berdoz, Luca A. Lanzendörfer, Fabian Farestam, Roger Wattenhofer · 2026
Read Reasoning Structure of Large Language Models by Frédéric Berdoz, Luca A. Lanzendörfer, Fabian Farestam, Roger Wattenhofer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03883.txt.
Introduction
Berdoz and co-authors turn reasoning traces into graphs so model thinking can be measured. Reasoning models are usually scored by final accuracy or token count. Equal scores can hide very different reasoning structures. A scalable benchmark of logic puzzles anchors the study. A pipeline converts raw traces into reasoning graphs. Claims and their dependencies become nodes and edges.
Reasoning becomes a structured object with a topology to analyze. A reasoning efficiency metric gauges how concentrated the logical flow is. Tests on open-source reasoning models show structural measures split behaviors that accuracy merges. Failure modes become easier to diagnose from the graphs. Scaling with puzzle difficulty can be compared across models in structural terms. Evaluation teams get a sharper lens than token counts. The graphs make reasoning quality inspectable, not just countable.
Catalog record
| CREATOR | Frédéric Berdoz, Luca A. Lanzendörfer, Fabian Farestam, Roger Wattenhofer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 120 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03883
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reasoning Structure of Large Language Models Frédéric Berdoz, Luca A. Lanzendörfer, Fabian Farestam, Roger Wattenhofer Abstract Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count. However, identical scores on these metrics can hide fundamentally different reasoning structures. To address this limitation, we introduce a scalable LRM benchmark of logic puzzles and a pipeline that converts unstructured traces into verifiable reasoning graphs of claims and dependencies. This turns reasoning into a structured, measurable object whose topology can be quantitatively analyzed. Building on this, we define a reasoning efficiency metric that quantifies how concentrated the model’s logical flow is. Our analysis on open-source reasoning models shows that structural measurements separate behaviors that token count and accuracy conflate, providing a practical tool for diagnosing failure modes and comparing how reasoning scales with puzzle difficulty. † † affiliationnotice: Code available at https://github.com/ETH-DISCO/llm-reasoning-efficiency . \eta=0.059 \eta=0.362 Figure 1 : Qualitative reasoning graphs. Shown are two extracted claim graphs from two independent samples of the same model (Qwen3 235B) on the same Tents puzzle instance: a diffuse trace (left) and a focused trace (right). Both traces solve the instance. Nodes are atomic claims and edges indicate deductive dependencies. Node color encodes claim validity: green = verified correct, red = verified wrong, grey = unverifiable, orange = tentative. Node opacity encodes claim status: opaque = stated, semi-transparent = derived. Accuracy and token count alone are too coarse to provide meaningful feedback about how the model reasoned, whereas graph structure and efficiency (annotated \eta ) distinguish diffuse exploration from focused deductions. 1 Introduction Reasoning is central to human intelligence and remains a major challenge for machine learning systems. Thanks to their ability to exploit increased test-time compute through long Chain-of-Thought (CoT) traces ( 46 ) , Large Reasoning Models (LRMs) have shown impressive performance on a broad set of reasoning tasks, including complex coding ( 5 ) , logical deduction ( 24 ) , mathematical reasoning ( 7 ) , and spatial reasoning ( 2 ) . However, because most evaluations collapse behavior into one-dimensional metrics such as final-answer accuracy or token count, it remains unclear how these models reason. This gap has motivated prior work to develop controllable puzzle environments that are less prone to benchmark saturation and data contamination ( 4 ; 54 ) . Logic puzzles have long attracted human curiosity because they are “easy to learn, but hard to master.” Unlike many real-world tasks, they are fully specified and admit unambiguous verification. Their difficulty can be scaled without changing the underlying rules, making them an ideal controlled benchmark for studying reasoning behavior. For example, 37 analyzed four puzzle families with adjustable complexity and found sharp regime changes and eventual collapse beyond a critical difficulty threshold. The authors also reported trace-level phenomena including overthinking, early fixation on incorrect hypotheses, and counterintuitive reductions in “reasoning effort” near failure ( 37 ) . Together, these results suggest that controllable puzzles can reveal failure modes that accuracy alone misses, and motivate analyses that use the intermediate traces rather than only the final answer. There is a tree at (4,1) and since it is in an edge, the only possible tent placements for the tree are (3,1) and (4,2). However column 2 has 0 tents, so (4,2) has no tent and (3,1) has a tent. The only adjacent cells to the tree at (1,2) are (1,1), (2,2) and (1,3). Note that column 2 has no tents so (2,2) is no tent. Column one can only have 1 tent and we have already placed it, so (1,1) is no tent. Thus (1,3) has a tent. The adjacent cells to the tree at (2,4) are (1,4), (2,3) and (3,4). Note that there is already a tent at (1,3) and since (1,4) and (2,3) are adjacent, they cannot be tents. So the tent must be at (3,4). Tree (4,1) COL 2: 0 tents Tree (1,2) Tree (2,4) Candidates: (3,1), (4,2) No Tent (4,2) Candidates: (1,1), (2,2), (1,3) Candidates: (1,4), (2,3), (3,4) Tent (3,1) COL 1: 1 tent No Tent (2,2) No Tent (1,4) No Tent (2,3) No Tent (1,1) Tent (1,3) Tent (3,4) Puzzle instance (input) Reasoning trace Reasoning graph Figure 2 : Reasoning graph extraction. Example of a 4 \times 4 Tents instance (left), an excerpt of a human-generated reasoning trace (middle), and the corresponding verifiable claim graph extracted using our pipeline described in Section 3 (right). This illustrates how our pipeline turns free-form traces into structured objects that can be analyzed beyond accuracy and token count. In this work, we move from measuring how much a model thinks to measuring the structure of its reasoning. We introduce a scalable benchmark of 2D grid puzzles and convert free-form textual traces into verifiable reasoning graphs of claims and dependencies, turning reasoning into a structured object that supports quantitative analysis. Graph topology, including depth and the fraction of the graph that supports the final solution, captures differences between focused reasoning and diffuse exploration that token count and final accuracy fail to capture. Building on this representation, we define a reasoning-flow efficiency metric \eta that measures the concentration of the model’s logical flow between its observations and its proposed solution. This metric can differentiate reasoning traces even when they reach the same correct solution with similar trace length. As Figure 6 shows, \eta exposes differences in reasoning structure between models even in regimes where accuracy is saturated and token budgets overlap. Analyzing reasoning traces, we find that token count is a suboptimal proxy for reasoning quality, since extra tokens primarily translate into verification overhead. In comparison, our proposed metric is able to track solution-relevant structure and correctness more effectively. Furthermore, our benchmark shows that even as models spend considerably more tokens at higher difficulty levels the hardest regime remains largely unsolved, indicating current limitations that are not resolved by simply allocating more compute. We summarize our contributions as follows: • We introduce a scalable benchmark of 21 logic puzzles that enables controlled scaling studies of LLM reasoning. • We convert free-form textual reasoning traces into verifiable reasoning graphs of claims and dependencies, making reasoning topology measurable beyond accuracy and token count metrics. • We define a reasoning efficiency metric \eta that separates focused reasoning from diffuse exploration by measuring how concentrated logical flow is relative to the minimal claim set. 2 Related Work 2.1 Eliciting Reasoning in LLMs Prompting, search, and RL at test time. Chain-of-Thought prompting elicits intermediate reasoning steps in language models ( 46 ) . Follow-up work adds explicit test-time search, including Tree-of-Thoughts with branching and backtracking ( 50 ) and Graph-of-Thoughts ( 3 ) , with adaptive variants that build task-specific DAG decompositions ( 32 ) . In parallel, reinforcement learning with verifiable rewards (RLVR) uses verifiable signals (e.g., unit tests or environment validators) to improve deliberation while reducing reward hacking concerns ( 22 ; 1 ) , and has enabled strong reasoning performance in recent systems ( 30 ; 13 ) . Rather than proposing new elicitation mechanisms, we analyze the structures these approaches produce by reconstructing verifiable trees from traces and studying how topology changes with puzzle complexity. 2.2 Benchmarking Reasoning Figure 3 : Puzzle suite overview. We evaluate models on