The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; -; - ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct Terms fo

Xue Wen Tan, Nathaniel Tan, Galen Lee, Stanley Kok · 2025

Read The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; -; - ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct Terms fo by Xue Wen Tan, Nathaniel Tan, Galen Lee, Stanley Kok (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20665.txt.

Introduction

Xue Wen Tan, Nathaniel Tan, Galen Lee, and Stanley Kok propose judging the quality of reasoning traces from large language models by their shape rather than by costly human scoring. Current checks rely on expert rubrics, slow hand labels, or pairwise votes, and simpler graph scores miss what good reasoning looks like.

Their method uses tools from topology, the study of shape in data, to extract features from the path a reasoning trace takes. In tests, these shape features predict trace quality far better than standard graph measures, and a small stable feature set does the job well. The result hints that strong reasoning has a rich geometric form and gives a cheap signal for training future models.

Catalog record

CREATORXue Wen Tan, Nathaniel Tan, Galen Lee, Stanley Kok
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20665

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; -; - ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper CCS: Do Not Use This Code Generate the Correct Terms for Your Paper Xue Wen Tan, Nathaniel Tan, Galen Lee, Stanley Kok Abstract . Evaluating the quality of reasoning traces from large language models remains understudied, labor-intensive, and unreliable: current practice relies on expert rubrics, manual annotation, and slow pairwise judgments. Automated efforts are dominated by graph-based proxies that quantify structural connectivity but do not clarify what constitutes high-quality reasoning; such abstractions can be overly simplistic for inherently complex processes. We introduce a topological data analysis (TDA)–based evaluation framework that captures the geometry of reasoning traces and enables label-efficient, automated assessment. In our empirical study, topological features yield substantially higher predictive power for assessing reasoning quality than standard graph metrics, suggesting that effective reasoning is better captured by higher-dimensional geometric structures rather than purely relational graphs. We further show that a compact, stable set of topological features reliably indicates trace quality, offering a practical signal for future reinforcement learning algorithms. 1. Introduction Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains and tasks, including commonsense reasoning, mathematical reasoning, logical reasoning, causal reasoning, visual reasoning, audio reasoning, multimodal reasoning, embodied reasoning, and defeasible reasoning ( Sun et al. 2023 ) . However, despite their impressive performance, the internal mechanisms underlying their reasoning processes remain poorly understood. Intermediate reasoning steps are often overlooked and treated merely as a means to the final answer rather than as objects of study in their own right. For instance, Wang et al. 2022 marginalizes diverse reasoning paths by aggregating multiple solutions and selecting the most consistent final answer, thereby discarding valuable information about the reasoning process itself. Moreover, there are growing concerns about the faithfulness of intermediate reasoning steps to the final answer ( Agarwal et al. 2024 ) . LLMs may arrive at correct answers through flawed or spurious reasoning processes, raising questions about whether the generated reasoning genuinely reflects the model’s decision-making or simply provides post-hoc justification for memorized answers. Evaluating the quality of reasoning processes presents significant challenges. First, there is a scarcity of datasets that provide step-by-step solutions; most datasets offer only final answers, forcing researchers to use answer correctness as an imperfect proxy for reasoning quality. Second, assessing reasoning quality is inherently subjective, and developing reliable quantitative metrics remains a complex and unresolved problem. To address these challenges, we introduce a novel evaluation framework grounded in a carefully curated dataset derived from the American Invitational Mathematics Examination (AIME) available on the Art of Problem Solving portal ( Art of Problem Solving 2025 ) . This dataset includes detailed solution steps contributed by mathematicians in the community. Importantly, we incorporate multiple solution paths for each problem, recognizing that mathematical problems can be solved through diverse reasoning strategies. However, we acknowledge that even expert-written solutions represent only the externalized steps of reasoning, omitting the intermediate cognitive processes required to progress from one step to the next. To capture a fuller picture of the reasoning process, we adapt the Smith-Waterman algorithm from biological sequence alignment ( Smith et al. 1981 ) to align LLM-generated reasoning traces with reference solutions, addressing the first challenge of dataset scarcity. To address the second challenge of subjective assessment, we employ topological data analysis (TDA) to quantify reasoning quality objectively. We hypothesize that high-quality reasoning traces exhibit distinctive structural characteristics that persist across diverse solution strategies. Topology provides a principled framework for capturing these invariant properties: just as a coffee mug and a donut are topologically equivalent (homeomorphic) despite their apparent differences ( Gamelin and Greene 1999 ) , diverse valid reasoning paths may share fundamental structural similarities that distinguish them from flawed reasoning. TDA enables us to identify and measure these invariant features, providing objective metrics for reasoning quality that are robust to surface-level variations in problem-solving approaches. Overall, our contributions include: • A novel application of the Smith-Waterman algorithm to align and compare LLM reasoning traces with expert solutions, enabling systematic evaluation on curated AIME problems with multiple solution paths. • A topological data analysis framework for objectively quantifying reasoning quality. • Empirical evidence that topological features provide discriminative signals for distinguishing high-quality from flawed reasoning traces. 2. Related Work 2.1. Reasoning Traces Analysis The analysis of reasoning traces in large language models (LLMs) has become a central research direction following the introduction of chain-of-thought (CoT) prompting ( Wei et al. 2022 ) , which improved problem-solving performance by encouraging models to produce intermediate reasoning steps. However, Self-Consistency prompting ( Wang et al. 2022 ) and related methods treat reasoning traces as instrumental by aggregating multiple reasoning paths to obtain the most frequent final answer while discarding the traces themselves. Recent work has shifted attention to reasoning faithfulness: whether a model’s explicit reasoning aligns with its underlying decision process. Agarwal et al. 2024 demonstrated that LLMs often produce plausible but unfaithful explanations that do not reflect actual reasoning, and Nguyen et al. 2024 found that models can achieve correct answers through logically flawed or spurious reasoning. These findings emphasize the need for metrics that evaluate reasoning process quality rather than final-answer correctness. To quantify reasoning structure, Xiong et al. 2025 introduced a reasoning-graph framework that maps LLM-generated reasoning traces into directed graphs, enabling analysis of properties such as branching and convergence. Their findings suggest that effective reasoning depends on a balanced pattern of exploration and convergence rather than on trace length alone, aligning with the observations of Su et al. 2025 that overly long reasoning often reduces accuracy. Complementary work has explored automated reasoning evaluation: Ton et al. [n. d.] proposed an information-theoretic approach measuring the contribution of each reasoning step, while Nguyen et al. 2024 grounded CoT reasoning in knowledge graphs to test factual and logical validity. Together, these works advance the field toward systematic evaluation of reasoning traces. Efforts to improve reasoning quality have paralleled those to analyze it. Reinforcement learning methods, such as GraphPRM ( Peng et al. 2025 ) , directly optimize reasoning steps rather than outcomes, using process-level rewards to align model behavior with logical norms. Structured prompting techniques such as Tree-of-Thoughts ( Yao et al. 2023 ) and Graph-of-Thoughts ( Besta et al. 2024 ) further demonstrate that the organization of reasoning whether tree, graph, or chain-based may critically influences problem-solving success. Collectively, these studies motivate the search for more robust and interpretable reasoning metrics, which our work seeks to advance through a geometric–topological formulation. 2.2. Topological Data Analysis Applied to LLMs Topological Data Analysis (TDA) offers a mathematical lens for understanding the shape of data by identifying invariant geometric structures such as connected components and holes. Recent work has applied TDA to study neural representations in large models. Gardinazzi et al. [n. d.] introduced zigzag persistent homology to analyze the evolution of topological features across

Read the complete text