LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinking Chünhung Wu, Jinliang Lu, Zixuan Ren, Gangqiang Hu, Zhi Wu, Dai Dai, Hua Wu Abstract Human cognition naturally engages with abstract and fluid concepts, whereas existing reasoning models often rely on generating discrete tokens, potentially constraining their expressive capabilities. Recent advancements aim to address this limitation by enabling large language models (LLMs) to generate soft, abstract tokens, thus facilitating reasoning within a continuous concept space. This paper explores the ‘Soft Thinking’ capabilities of various LLMs by examining the models’ internal behavior using a suite of probing techniques. Contrary to the common belief that Soft Thinking enables the simultaneous exploration of diverse reasoning paths, our findings reveal that LLMs are single-threaded reasoners. They predominantly rely on the single token with the highest probability in the soft inputs to predict the next step. This reliance hinders the exploration of different reasoning paths and reduces vanilla Soft Thinking to a form of greedy decoding, obscuring the advantage of transmitting more information through Soft Tokens. To tackle this issue, we explore sampling strategies to introduce randomness , employing methods such as Dirichlet resampling and the Gumbel-Softmax trick. Our experiments demonstrate that incorporating randomness can alleviate the limitations of vanilla approaches and unleash the potential of Soft Thinking. Notably, the Gumbel-Softmax trick provides adequate randomness with controlled smoothness, resulting in superior performance across eight reasoning benchmarks. 1 Introduction Large Language Models (LLMs) have made significant advancements across a wide array of tasks ( Jaech et al. 2024 ; DeepSeek-AI et al. 2025 ) . A key technique underpinning these improvements is Chain-of-Thought (CoT) reasoning, in which models tackle complex problems by generating intermediate reasoning steps in natural language ( Kojima et al. 2022 ) . Despite its success, the conventional CoT approach has inherent limitations. Specifically, it confines the model’s reasoning process to a sequence of predefined discrete tokens. This may fundamentally limit the model’s reasoning capability beyond natural language, causing the model to commit to certain solutions and avoid exploring alternatives ( Yao et al. 2023 ) . Unlike LLMs, humans can reason with abstract concepts. Neuroscientific studies have shown that many aspects of thought engage distinct brain regions from, and do not depend on, language ( Fedorenko & Varley 2016 ; Benn et al. 2023 ; Fedorenko et al. 2024 ) . Drawing inspiration from these cognitive insights, previous research has aimed to enhance models’ abilities to reason within a continuous space, leading to the development of various Soft Thinking (Latent CoT) approaches ( Hao et al. 2024 ; Shen et al. 2025 ; Zhang et al. 2025 ) . These papers replace the discrete tokens with hidden states or output probabilities, enabling LLMs to reason with soft, abstract concepts. They hope that the models can transmit more information with Soft Tokens and explore multiple potential reasoning paths simultaneously ( Hao et al. 2024 ; Zhang et al. 2025 ; Gozeten et al. 2025 ) . While performance gains are reported in these papers, the working mechanism of Soft Thinking remains to be clarified. In this paper, we explore the Soft Thinking ability of modern LLMs with strong reasoning abilities ( DeepSeek-AI et al. 2025 ; Team 2025 ; He et al. 2025 ) . In our preliminary experiments, we observed some unexpected outcomes: the vanilla Soft Thinking approach performs significantly worse than traditional token sampling decoding methods. To explore the reasons behind this phenomenon, we examined the model’s internal behavior using a suite of probing techniques. Our findings suggest that the popular belief that Soft Thinking enables parallel exploration of multiple reasoning paths is not supported. As shown in Fig 1 , LLMs predominantly rely on the single token with the highest probability in the soft inputs to predict the next step. This creates a feedback loop that favors the most self-assured reasoning path, hindering the exploration of alternative reasoning paths. While transmitting more information through Soft Tokens indeed creates novel reasoning paths, the inherent greedy tendency prevents the LLM from fully capitalizing on this advantage. To circumvent this Greedy Pitfall, we explored various methods to restore randomness into Soft Thinking, including Dirichlet Sampling and the Gumbel-Softmax trick ( Gumbel 1954 ; Maddison et al. 2014 ; Kool et al. 2019 ) . Empirical evaluations conducted on challenge reasoning benchmarks using three mainstream reasoning LLMs demonstrate that incorporating randomness can mitigate the greedy tendencies of Soft Thinking. Furthermore, the Gumbel-Softmax trick, which provides both adequate randomness and flexible control over smoothness through the temperature hyperparameter, can further unleash the potential of Soft Thinking, achieving consistent performance improvements across a wide range of tasks. Figure 1: Left: Soft Thinking replaces the discrete token t with the Soft Token st (defined as the probability distribution over vocabulary), enabling LLMs to reason in continuous concept space. Right: An illustration of the latent search tree of Soft Thinking. Soft Thinking predominantly explores sequences associated with the highest probability token. In contrast, paths stemming from non-top-1 tokens are typically terminated in the next step. 2 Backgrounds: Discrete Thinking and Soft Thinking 2.1 Discrete Thinking We begin by outlining the discrete token generation process in LLMs. Given a query \mathbf{x} , the model generates a sequence \mathbf{y} , which includes both the intermediate reasoning path \mathbf{t} and the final answer \mathbf{a} . Initially, the model generates the intermediate reasoning path \mathbf{t} . Upon decoding a special end-of-thinking token, , the model transitions to generating the final answer \mathbf{a} . During each decoding step, the LLM uses the input \mathbf{x} and all previously generated tokens \mathbf{y}_{), the reasoning process is terminated, and the model switches to discrete token decoding to generate a readable final answer. 2.3 Possible Working Mechanism of Soft Thinking One of the most intriguing properties of Soft Thinking is its potential to inherently preserve multiple reasoning paths, effectively forming a latent search tree. COCONUT suggested that models utilizing Soft Thinking can maintain a diverse set of possibilities within the continuous reasoning process ( Hao et al. 2024 ) . Soft Thinking ( Zhang et al. 2025 ) and CoT2 ( Gozeten et al. 2025 ) offer theoretical frameworks demonstrating how LLMs may approximate the comprehensive path-summation of standard CoT by iteratively constructing linear surrogate representations. However, the evidence supporting these hypotheses is still lacking. 3 Preliminary Experiments: Is Soft Thinking Effective? In this section, we present our initial experiments with a vanilla implementation of the Soft Thinking approach. We begin by detailing the tasks and LLMs employed in our experiments. Subsequently, we demonstrate that vanilla Soft Thinking generally underperforms compared to token CoT. 3.1 Experiment Setting Models. We utilize a range of mainstream LLMs known for their robust reasoning capabilities. These include Deepseek-R1-Distill-Qwen-32B ( DeepSeek-AI et al. 2025 ) , QwQ-32B ( Team 2025 ) , Skywork-OR1-32B ( He et al. 2025 ) . Benchmarks. Our evaluation is conducted on seven diverse benchmarks. These include AIME’24/’25 ( Mathematical Association of America 2024 ) , MATH-500 ( Cobbe et al. 2021 ) , and AMC’23 in the mathematical domain, GPQA-Diamond ( Rein et al. 2023 ) for knowledge-based question-and-answer tasks, and HumanEval ( Chen et al. 2021 ) , MBPP ( Austin et al. 2021 ) , and LiveCodeBench ( Jain et al. 2025 ) for code generation. This comprehensive suite allows us to thoroughly assess the performance across varied domains and tasks. Implementation Details. We utilize the SGLang ( Zheng et al. 2024 ) Soft Thinking implementation provided by Zhang et al. 2025 . For both vanilla Soft Thinking and discrete Token Thinking, we set the temperature to 0.6, top- p to 0.95, and top- k to 30. Across all experiments, the maximum generation length is capped at 32,768 tokens. For the AIME’24/’25 and AMC’23 benchmarks, we report Average@32 to mitigate the effects of randomness in smaller test sets. For the remaining benchmarks, we report Pass@1 to evaluate performance. Table 1: Detailed results on math, knowledge Q&A, and code benchmarks. Results highlighted in green indicate an improvement or performance comparable to Token CoT. Results highlighted in red signal a decline in performance relative to Token CoT. It is shown that Vanilla Soft Thinking generally leads to lower performance, which is similar to greedy decoding. Thinking Mode AIME24 AIME25 MATH500 AMC23 GPQA-Diamond HumanEval MBPP LiveCodeBench Avg Deepseek-R1-Distill-Qwen-32B Token (Greedy) 66.66 50.00 92.20 85.00 60.10 87.20 88.71 42.65 71.57 Token (Sampling) 72.08 55.63 94.50 95.46 60.60 97.25 95.13 57.35 78.50 Soft (Vanilla) 62.00 49.17 91.60 90.00 60.10 86.41 87.93 44.80 72.13 QwQ-32B Token (Greedy) 80.00 70.00 97.00 100.00 64.14 95.12 96.10 58.78 82.64 Token (Sampling) 77.92 67.50 96.20 97.50 62.63 98.17 96.89 62.00 82.35 Soft (Vanilla) 76.67 62.29 96.20 98.75 59.60 93.90 95.33 57.71 80.06 Skywork-OR1-32B Token (Greedy) 76.67 73.33 95.80 90.00 56.06 81.71 86.38 54.84 76.85 Token (Sampling) 78.75 71.25 96.40 98.28 62.62 96.95 97.28 62.37 82.99 Soft (Vanilla) 79.16 69.38 96.00 97.97 59.60 85.37 90.66 55.56 79.21 3.2 Experimental Results As shown in Table 1 , across all tested large language models (LLMs), vanilla Soft Thinking consistently underperforms compared to discrete Token Thinking. The only exception is the performance of the QwQ-32B model on mathematical benchmarks, where the greedy Token Thinking outperforms discrete Token Thinking, indicating potential overfitting on these benchmarks. Overall, we find that vanilla Soft Thinking does not lead to improvements compared to discrete Token thinking in both accuracy and efficiency. Instead, it appears to achieve similar performance with greedy decoding. Figure 2: An example illustrating the probability distribution of the vanilla Soft Thinking method. 4 Analysis of Soft Thinking Behavior Despite the theoretical promise of Soft Thinking to enhance the reasoning capabilities of large language models, our preliminary experiments revealed that its performance did not surpass—and in most cases even fell below—that of traditional token decoding methods. This unexpected result prompted us to question the actual behavior of Soft Thinking in practice: Why does incorporating vanilla Soft Thinking not lead to the anticipated performance improvements? To uncover the underlying reasons, we decided to delve deeper into the model’s behavior. 4.1 Case Study Fig. 2 presents a case study of the vanilla Soft Thinking process applied to a simple question. Through this example, we observe that, in consecutive decoding steps, the components of the subsequent token consistently and exclusively exhibit semantic coherence with the dominant token from the preceding step. For instance, in steps 2 and 3, the word ‘I’ is not a semantically appropriate successor to ‘let’. Similarly, the words in step 44 are capitalized, suggesting they can only logically follow a period token from step 43. Based on these observations, we propose the following hypothesis: Hypothesis: LLMs are Signle-Threaded Reasoners LLMs lack the ability to process multiple different semantic trajectories in parallel. When a Soft Token is fed into an LLM, the generation process is typically dominated by the majority component of the Soft Token, particularly when the Soft Token’s weight is highly concentrated. To verify this hypothesis, we employ QwQ-32B to generate responses for AIME’24 and AIME’25. We collect prediction distributions for nearly 10^{6} steps and only preserve the intermediate reasoning steps. Subsequently, we analyze the decoding behavior of these generated responses. 4.2 Output Probability of Soft Thinking (a) Soft Token entropy (b) 1st Token (c) 2nd Token Figure 3: Output entropy/token probability vs. JS-divergence between next token prediction probabilities yield from different inputs. The prediction of soft input is nearly identical to the prediction of the 1st token input but completely different from the prediction of the 2nd token input. We first analyze the model’s output probability in the identified soft steps by conducting three distinct forward passes and extracting their output probabilities: one with the entire Soft Token, P_{\mathbf{st}} , one with the highest probability token within the Soft Token, P_{1} , and another with the second highest probability token in the Soft Token set, P_{2} . We then compute the Jensen-Shannon (JS) Divergence to quantify the variation in prediction behavior resulting from these different inputs: JS(\mathbf{st},\mathbf{t}^{i})=\frac{1}{2}\left(KL(P_{\mathbf{st}}||Q_{i}))+KL(P_{i}||Q_{i})\right),\quad i=1,2 Here, KL(P||Q) denotes the KL divergence between probability distributions P and Q . Q_{i} is the mid-point distribution: Q_{i}=\frac{1}{2}(P_{\mathbf{st}}+P_{i}) . A near-zero JS divergence suggests that the model’s prediction behavior during a Soft Thinking step closely aligns with the outcome when considering just that particular token. This implies that the single token in the Soft Token set has a substantial influence on the model’s reasoning process during these steps. Conversely, a high JS divergence indicates that the token merely contributes to the Soft Thinking process, suggesting a more distributed or collaborative influence among multiple tokens. We plotted the JS divergence alongside the normalized entropy of the Soft Token and the probability of the first and second token. As suggested in prior works ( Zhang et al. 2025 ; Gozeten et al. 2025 ) , if the LLM is exploring different reasoning paths simultaneously, the final probability distribution for the Soft Token is expected to be a linear combination of individual token forward results: p(\mathbf{y}_{t}|\mathbf{st},\mathbf{x},\mathbf{y}_{