ThoughtProbe: Classifier-Guided Thought Space Exploration Leveraging LLMs’ Intrinsic Reasoning
Zijian Wang, Chang Xu · 2025
Read ThoughtProbe: Classifier-Guided Thought Space Exploration Leveraging LLMs’ Intrinsic Reasoning by Zijian Wang, Chang Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06650.txt.
Introduction
Zijian Wang and Chang Xu found in 2025 that a model's hidden states betray how well it is reasoning, and used that to steer its search. Big language models hold reasoning skill that surfaces when their answer space widens, but how to spend that skill was unclear. A plain linear classifier on the right layers and state types could spot strong reasoning in the making.
Their frame, ThoughtProbe, grows a tree of answers and lets the classifier score each branch, pouring compute into the most thoughtful lines. At the end, answers pool and a branch-sum rule picks the winner. Across arithmetic tests the method found and used good chains far better than plain sampling.
Catalog record
| CREATOR | Zijian Wang, Chang Xu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ThoughtProbe: Classifier-Guided Thought Space Exploration Leveraging LLMs’ Intrinsic Reasoning Zijian Wang, Chang Xu Abstract Pre-trained large language models (LLMs) have been demonstrated to possess intrinsic reasoning capabilities that can emerge naturally when expanding the response space. However, the neural representation mechanisms underlying these intrinsic capabilities and approaches for their optimal utilization remain inadequately understood. In this work, we make the key discovery that a simple linear classifier can effectively detect intrinsic reasoning capabilities in LLMs’ activation space, particularly within specific representation types and network layers. Based on this finding, we propose a classifier-guided search framework that strategically explore a tree-structured response space. In each node expansion, the classifier serves as a scoring and ranking mechanism that efficiently allocates computational resources by identifying and prioritizing more thoughtful reasoning directions for continuation. After completing the tree expansion, we collect answers from all branches to form a candidate answer pool. We propose a branch-aggregation selection method that marginalizes over all supporting branches by aggregating their thoughtfulness scores, thereby identifying the optimal answer from the pool. Experimental results show that our framework’s comprehensive exploration not only covers valid reasoning chains but also effectively identifies them, achieving significant improvements across multiple arithmetic reasoning benchmarks. 1 Introduction Pre-trained large language models (LLMs) often struggle to demonstrate robust reasoning capabilities, primarily due to their tendency to provide short intuitive responses rather than engaging in a deliberative reasoning process ( Rae et al., 2021 ) . This limitation makes them particularly prone to errors when tackling tasks that demand multi-step logical reasoning, such as mathematical arithmetic ( Yao et al., 2024 ; Wei et al., 2022 ) . Some works aim to enhance pre-trained LLMs’ reasoning through approaches that require human involvement - whether designing prompts or tuning with annotated data ( Zelikman et al., 2022 ; Hoffman et al., 2024 ; Zhang et al., 2024b ; Wei et al., 2022 ; Kojima et al., 2022 ) . However, these methods passively inject human reasoning prior knowledge into LLMs rather than actively exploring and leveraging their intrinsic reasoning capabilities. Figure 1: Pre-trained LLMs could naturally generate both thoughtful and non-thoughtful responses when sampling multiple times, with these responses being linearly separable in the activation space. Recent evidence suggests that pre-trained LLMs may possess intrinsic reasoning capabilities even without explicit human intervention. When sampling multiple responses with only a question as prompt, lengthy reasoning chains can emerge naturally and could lead to correct answers ( Wang & Zhou, 2024 ; Wang et al., 2022 ) , as shown in Figure 1 , indicating intrinsic reasoning abilities that single-pass inference fails to capture. Building on this observation, recent frameworks have successfully employed search algorithms to explore response space and identify effective reasoning chains ( Wu et al., 2024 ; Luo et al., 2024b ; Snell et al., 2024 ; Lightman et al., 2023 ; Wang et al., 2024 ) , using various guiding signals from semantic analysis ( Zhang et al., 2024a ) to LLM-based verbal evaluation ( Xie et al., 2024 ) . However, key questions remain unexplored: How are reasoning capabilities represented within LLMs’ internal activations, and how can these intrinsic abilities be effectively harnessed? Better understanding of these aspects could lead to more interpretable and reliable methods. To address this issue, our research aim to investigate pre-trained LLMs intrinsic reasoning capabilities at hidden representation level. We first introduce the concept of thoughtfulness - the degree to which a sampled response demonstrates deliberative reasoning content rather than short intuitive answers - as a proxy to study and leverage LLMs’ intrinsic reasoning capabilities. Our empirical evidence reveals that the Linear Representation Hypothesis (LRH) ( Zou et al., ; von Rütte et al., 2024 ) extends to thoughtfulness, showing that thoughtful and non-thoughtful responses are linearly separable in LLMs’ activation space, as shown in Figure 1 . Specifically, we find that this separability is most prominent in certain representation types and layer ranges, where a simple linear classifier can effectively distinguish thoughtful responses from non-thoughtful ones. Also, we show the classifier consistently assigns higher scores to correct, thoughtful responses and lower scores to incorrect ones, regardless of response length. Motivated by this discovery, we present ThoughtProbe, a classifier-guided inference-time computing framework that systematically leverages LLMs’ intrinsic thoughtfulness for effective response space exploration. Our framework is built on the intuition that thoughtful responses are more likely to contain correct reasoning steps, thus increasing the probability of finding valid solution paths within the search space. The exploration process is structured as an iterative tree search, with the initial question as the root node. At each step, multiple token sequences are generated as candidate child nodes, whose thoughtfulness score is evaluated by our classifier through probing hidden representations. By selecting more thoughtful candidates for continuation, we efficiently allocate computational resources and increase the likelihood of including correct reasoning trajectories in our exploration tree. This exploration process continues until either reaching the termination token or exhausting the computational budget. Upon completion of tree expansion, we obtain multiple branches, each leading to a candidate answer, forming a comprehensive answer pool. To determine the optimal one, we propose a branch-aggregation selection through value marginalization that considers thoughtfulness score across all branches leading to each candidate answer. Specifically, the value of each answer is computed by aggregating the thoughtfulness score of all its supporting branches, with the final answer selected as the one that achieves the highest marginal value. Experiments on multiple mathematical reasoning benchmarks demonstrate that our method consistently outperforms existing approaches, achieving significant improvements. Our work provides new insights into enhancing LLMs’ reasoning capabilities and opens up promising directions for developing more robust reasoning systems. 2 Preliminary 2.1 LLMs Architecture and Hidden Representation To provide a foundation for the discussion, we first describe the basic structure of a Transformer-based LLM architecture ( Vaswani, 2017 ) . The input text is initially tokenized into a sequence of tokens, which are then mapped to embeddings to form the initial representation sequence \mathbf{x}^{(0)}\in\mathbb{R}^{T\times d_{\text{emb}}} . Here, T is the sequence length, and d_{\text{emb}} is the embedding dimension. The embeddings are then processed through multiple Transformer layers. In each layer l , its hidden states are composed of three components: activations from multi-head self-attention (MHA), multi-layer perceptron (MLP), and residual connections. This process can be formulated as: \displaystyle\mathbf{a}^{(l)}_{\text{attn}} \displaystyle=\text{MHA}(\mathbf{h}^{(l)}) (Attention activations) \displaystyle\mathbf{a}^{(l)}_{\text{mlp}} \displaystyle=\text{MLP}(\mathbf{a}^{(l)}_{\text{attn}}+\mathbf{h}^{(l)}) (MLP activations) \displaystyle\mathbf{h}^{(l+1)} \displaystyle=\mathbf{a}^{(l)}_{\text{mlp}}+\mathbf{a}^{(l)}_{\text{attn}}+\mathbf{h}^{(l)} (Hidden states) 2.2 LLMs Reasoning Structure Reasoning structures typically manifest in two fundamental topologies: sequential chains and branching trees. The chain structure reflects the step-by-step nature of logical deduction, while the tree structure captures the exploration of multiple potential reasoning paths. Below, we formally define these structures and their probabilistic formulations. Reasoning Chain For an input question Q , a reasoning chain is defined as a sequence of intermediate thought steps R=[Q,r_{1},r_{2},...,r_{N}] , leading to a final answer A . Here, r_{i} represents an intermediate thought at the i -th step, and N denotes the chain length. The answer can be extracted by appending a trigger prompt at the end of the chain, like “Therefore, the answer is”. The probability of generating such a chain can be formalized as: P(R,A|Q)=P(r_{1}|Q)\prod_{i=2}^{N}P(r_{i}|Q,r_{1:i-1})\cdot P(A|Q,R) where P(r_{1}|Q) is the probability of the first step, P(r_{i}|Q,r_{1:i-1}) is