Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces
Sanjoy Chowdhury, Dinesh Manocha · 2026
Read Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces by Sanjoy Chowdhury, Dinesh Manocha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14358.txt.
Introduction
Chowdhury and Manocha measure how much of a language model's long chain-of-thought trace is actually needed to keep its answer. The minimal core is the smallest step subset preserving the answer or prediction distribution. On average, about 46 percent of steps prove removable while keeping the original answer in 86 percent of cases. Support is concentrated in the top three steps, carrying 65 percent of measured necessity. Minimal cores separate right from wrong traces better. They also transfer across model families, suggesting much trace text is verbose padding.
Catalog record
| CREATOR | Sanjoy Chowdhury, Dinesh Manocha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70277 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces Sanjoy Chowdhury, Dinesh Manocha Abstract Language models often generate long chain-of-thought traces, but it remains unclear how much of this reasoning is necessary for preserving the final prediction. We study this through the lens of overcomplete reasoning traces : generated traces that contain more intermediate steps than are needed to support the model’s answer. We define the minimal core as the smallest subset of steps that preserves either the final answer or predictive distribution, and introduce metrics for compression ratio, redundancy mass, step necessity, and necessity concentration. Across six deliberative reasoning benchmarks spanning arithmetic, competition mathematics, expert scientific reasoning, and commonsense multi-hop QA, we find substantial overcompleteness: on average, 46% of steps are removable under greedy minimal-core extraction while preserving the original answer in 86% of cases. We also find that predictive support is concentrated: the top three steps account for 65% of measured necessity mass on average. Beyond compression, minimal cores expose a cleaner geometry of reasoning: compared with full traces, they improve correct–incorrect trace separation by 11 points , reduce estimated intrinsic dimensionality by 34% , and transfer across model families with 85% off-diagonal answer retention. Theoretically, we establish existence of minimal sufficient subsets, local irreducibility guarantees for greedy elimination, and certificates of overcompleteness and sparse necessity. Together, these results suggest that full reasoning traces are often verbose and overcomplete, while minimal cores isolate the effective support underlying language-model predictions. 1 Introduction Language models increasingly use explicit intermediate reasoning traces such as chain-of-thought derivations, plan-style decompositions, and step-by-step explanations to solve complex reasoning problems ( 66 ; 32 ; 78 ; 70 ) . These traces are now central to prompting, verification, process supervision, and evaluation ( 64 ; 39 ) . Yet a generated trace is not necessarily a minimal explanation 56 ; 19 : it may include restatements, paraphrases, stylistic justifications, redundant checks, or alternative routes that do not affect the final prediction. Fig. 1 summarizes our view: full traces often contain a compact predictive core surrounded by removable reasoning mass. Figure 1 : LLM reasoning traces are substantially overcomplete. Generated traces contain a sparse predictive core surrounded by removable reasoning mass. Minimal-core extraction separates answer-supporting steps from redundant elaboration and exposes a compact, transferable representation geometry of reasoning. This raises a simple question: how much of a generated reasoning trace is actually needed? Prior work has largely focused on eliciting better reasoning 21 ; 47 ; 43 ; 24 , supervising intermediate steps 68 ; 44 ; 31 ; 76 , improving self-consistency 63 ; 79 ; 58 ; 74 , or training models to reason more efficiently ( 66 ; 64 ; 39 ; 73 ) . Recent efficient-reasoning methods further show that shorter traces can sometimes preserve or improve accuracy through curated data, reinforcement learning, test-time scaling 62 ; 60 ; 69 ; 55 , or trimming objectives ( 71 ; 46 ; 54 ; 65 ; 6 ) . These directions are complementary to ours: rather than making a model generate shorter traces, we analyze traces that have already been generated and ask which steps are behaviorally sufficient for preserving the original answer or predictive distribution. We study this problem through the lens of overcomplete reasoning traces . A trace is overcomplete if it contains a proper subset of steps that preserves the model’s predictive behavior. We define the minimal core as the smallest such sufficient subset. This turns trace analysis into a subset-sufficiency problem and induces quantitative measures of compression ratio, redundancy mass, step necessity, and necessity concentration. Unlike human-rationale extraction, our goal is not to recover a canonical explanation; instead, we characterize the effective support of a model prediction under explicit reasoning context. Minimal cores are useful for both interpretation and evaluation. Accuracy indicates whether the final answer is correct, but not which parts of a trace support that answer. Full-trace evaluation can conflate essential reasoning with removable elaboration. Minimal cores instead expose the effective support of a prediction, allowing us to test whether reasoning utility is distributed across many steps or concentrated in a sparse set of high-necessity steps. Beyond compression, we ask whether minimal cores reveal a cleaner representation geometry of reasoning, by which we mean the compactness, correct–incorrect separability, intrinsic dimensionality, and transfer behavior of trace embeddings. Representation-analysis work shows that neural embeddings can encode linguistic structure, task information, truthfulness, uncertainty, and latent decision states ( 33 ; 27 ; 3 ; 1 ; 35 ; 45 ) . We extend this perspective to generated reasoning traces: if full traces mix answer-supporting reasoning with redundant stylistic variation, then minimal-core embeddings should be more compact, lower-dimensional, and more informative about correctness than full-trace embeddings. We evaluate this framework on six deliberative reasoning benchmarks spanning arithmetic, competition mathematics, expert scientific reasoning, and commonsense multi-hop QA: GSM8K ( 15 ) , MATH500 ( 25 ; 39 ) , AIME24, AMC23, GPQA-Diamond ( 49 ) , and StrategyQA ( 23 ) . Across model families, greedy minimal-core extraction removes nearly half of generated steps while largely preserving the full-trace answer. Predictive support is concentrated in a small number of steps, minimal cores transfer across model families, and their embeddings are more compact and correctness-informative than full-trace or removed-step embeddings. We make the following key contributions: • Trace-level minimal cores. We study minimal sufficiency over generated reasoning traces, defining minimal cores as the effective support of a prediction. • Necessity and overcompleteness. We quantify removable reasoning through redundancy mass, step necessity, and necessity concentration. • Guarantees for core extraction. We establish guarantees for existence, local minimality under greedy deletion, overcompleteness certificates, and sparse-necessity structure. • Transferable and geometric support. We show that minimal cores transfer across model families and occupy more compact, correctness-informative regions of representation space. 2 Minimal Cores in Overcomplete Reasoning Traces We view generated reasoning traces as structured objects containing both answer-supporting steps and redundant elaboration. A full chain-of-thought trace may include restatements, paraphrases, checks, and stylistic explanations that improve readability but are not necessarily required for preserving the model’s prediction. We formalize this by defining the minimal core : a smallest subset of generated steps that preserves the predictive behavior of the original trace. Let x be the input and M a language model. Under a prompting strategy such as chain-of-thought or plan-then-solve, the model produces a trace followed by a final answer. We write the trace as z=(r_{1},\ldots,r_{T}) , where each r_{t} is a textual reasoning unit obtained from numbered model outputs or a fixed deterministic segmentation rule. For any subset S\subseteq[T] , let z_{S}=(r_{t}:t\in S) denote the retained subsequence in original order. Conditioning on x and z_{S} , the model induces p_{M}(\cdot\mid x,z_{S}) and prediction \hat{y}_{M}(x,z_{S})=\arg\max_{y}p_{M}(y\mid x,z_{S}), (1) with full-trace prediction \hat{y}_{M}(x,z_{[T]}) . Our goal is behavioral rather than mechanistic: we do not claim that deleting written steps reveals the model’s exact internal computation. Instead, we ask which generated steps are functionally necessary for preserving the prediction when the trace is used as explicit reasoning context, giving a post-hoc way to analyze effective support without human rationales or retraining. A retained subset can be sufficient at different strictness levels. Answer-preserving sufficiency requires only the same