Lexical hints of accuracy in LLM reasoning chains Authors: Arne Vanhoyweghen, Brecht Verbeken, Andres Algaba, Vincent Ginis Published: 2026 ABSTRACT Fine-tuning Large Language Models (LLMs) with reinforcement learning to produce explicit Chain-of-Thought (CoT) reasoning improves performance across code, math, and general-knowledge benchmarks. However, on tasks where accuracy remains low, such as Humanity’s Last Exam (HLE), models often express high self-confidence despite incorrect answers, revealing poor calibration. We investigate whether measurable properties of the CoT offer reliable, model-internal confidence signals. We analyze three feature classes: (i) CoT length, (ii) intra-CoT sentiment volatility, and (iii) lexicographic markers, including hedging terms. Using DeepSeek-R1, Claude 3.7 Sonnet, and Qwen-235B-Think, we evaluate these features on HLE, Omni-MATH, and GPQA-diamond. Across benchmarks, lexical uncertainty cues (e.g., guess, stuck, hard) are among the most informative indicators, while sentiment shifts provide a weaker but complementary signal. CoT length is informative only on Omni-MATH and GPQA, where accuracy is already high (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 70\%$$\end{document}), and carries no signal on the harder HLE (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 9\%$$\end{document}), indicating that CoT length predicts correctness only in the intermediate-difficulty benchmarks, i.e., inside the model’s demonstrated capability, but still below saturation. Finally, uncertainty signals in the CoT are more salient than high-confidence markers, making errors easier to detect than correct responses. These findings support a lightweight post-hoc calibration method that complements unreliable self-reported probabilities. Introduction Large Language Models (LLMs) can be fine-tuned with reinforcement learning to produce a Chain-of-Thought (CoT) before delivering their final response to improve their general performance1–4. The fine-tuning rewards are based both on the final correctness of the response—to improve capabilities—and on adherence to the CoT format, which may, for example, include safety-related specifications to enhance readability and alignment5,6. Although LLMs have shown a strong performance increase on demanding benchmarks, such as GPQA7, SWE-bench8, and FrontierMath9, they often report very high confidence while still obtaining relatively low overall accuracy, reflecting poor calibration10–12. This miscalibration masks silent failure modes and undermines the reliability of LLMs in open-ended settings13,14. In addition to the poorly calibrated self-reported confidence11, the confidence of LLMs in their final responses can be assessed via sampling agreement techniques that are computationally expensive15–17, or by monitoring internal representations, which requires access to the model weights18–20. In contrast, the CoT of some LLMs is readily available with its final response, offering a potential proxy for assessing the model’s internal confidence21,22. Although it has been shown that an LLM’s CoT may not always faithfully reflect the model’s reasoning process and may be difficult to interpret2,23–25, there is some evidence that specific CoT properties predict the accuracy of the final response26,27. For example, there is mixed evidence on how informative the length of the CoT is for predicting response accuracy28–32. Beyond quantitative properties such as length, recent studies have investigated how linguistic expressions of uncertainty relate to model correctness. For instance,33 show that direct answers from LLMs tend to avoid epistemic markers altogether and, when they do appear, often overuse strengtheners—resulting in overconfident yet incorrect generations. More recent work has demonstrated that such linguistic cues can even be manipulated to affect model calibration34. In contrast, our analysis focuses on the chain-of-thought (CoT) of reasoning models, systematically quantifying naturally occurring lexical uncertainty markers across benchmarks and model families. In this paper, we investigate whether Chain-of-Thought provides a reliable proxy for an LLM’s internal confidence by systematically analysing three classes of features: (i) CoT length, aligning with prior work, (ii) intra-CoT sentiment volatility, and (iii) lexicographic indicators, such as hedging words. Whereas most existing studies focus narrowly on length-based correlations, our contribution is to extend this analysis with sentiment and lexical markers. We evaluate these features on three contrasting benchmarks: Omni-MATH, where state-of-the-art models already perform well; Humanity’s Last Exam (HLE), which remains highly challenging; and GPQA-Diamond, a graduate-level reasoning benchmark designed to test conceptual depth and domain generalisation. The results show that CoT length and sentiment dynamics provide moderate predictive power in the “easier” benchmarks Omni-MATH and GPQA, but offer little signal on HLE. In contrast, lexical cues consistently yield informative calibration signals across all tasks. Methods Datasets and evaluation To analyse the lexicographic and linguistic characteristics of LLM reasoning, we collect the CoT and final responses from DeepSeek-R16, Claude Sonnet 3.735, and Qwen-235B-Think36 on three large-scale benchmarks of varying difficulty: Omni-MATH37, GPQA38, and HLE11. We choose DeepSeek-R1, Claude 3.7 Sonnet (version: 20250219), and Qwen-235B-Think because their complete CoTs are fully available, and the serial test-time compute protocol that produces them—including the configurable maximum-token cap—is publicly documented6,36,39. Although the two models achieve comparable benchmark scores, Claude 3.7 Sonnet employs more extensive alignment techniques, such as constitutional fine-tuning40, which may affect its calibration and the uncertainty signals visible in its CoT41. Starting from the original 3, 000-question HLE dataset11, we first remove 316 multimodal items. From the remaining 2, 684 questions, we then exclude 412 multiple-choice items, resulting in 2, 088 open-ended questions used in our analysis. The official benchmark results report that DeepSeek-R1, Claude Sonnet 3.7, and Qwen-235B-Think achieve an accuracy of 8.6% (calibration error: 81.4%), 8.9% (calibration error: 88.3%), and 11.8% (calibration error: 74.0%), respectively, on the full (text-only for DeepSeek-R1) version of the benchmark42. Omni-MATH comprises 4, 428 mathematics problems ranging from basic arithmetic to advanced algebra and geometry. On this benchmark, reasoning models, such as OpenAI o3-mini, achieve over 70% accuracy and even surpass 80% accuracy for algebra and calculus in a high reasoning effort setting28. Finally, we include GPQA-Diamond38, a multiple-choice, graduate-level scientific reasoning benchmark covering physics, chemistry, and biology (n=198 questions). GPQA-Diamond represents the most challenging tier of the GPQA suite, with official benchmark accuracies of 70.8% for DeepSeek-R1, 77.2% for Claude Sonnet 3.7, and 79.9% for Qwen-235B-Think. In contrast to the other models, Qwen-235B-Think exhibited an unusually high failure rate in producing valid answers in the required format. After a first pass (e.g., on Omnimath), fewer than 10% of the questions were returned with an interpretable answer, in sharp imbalance with the other models. To mitigate this, we implemented a rerun policy: each missing response from Qwen-235B-Think was resubmitted up to 10 times. Table 1 reflects this adjustment, where missing answers and confidence scores were filled in under this rerun procedure. Even after this intervention, Qwen-235B-Think remains the model with the largest proportion of missing answers. Omni-MATH, HLE, and GPQA-Diamond represent three complementary benchmarks that jointly span the reasoning spectrum. Omni-MATH offers 4,428 tiered mathematics problems with automatic, high-fidelity grading37, allowing us to observe CoT behaviour across a continuous, well-calibrated difficulty spectrum. In contrast, HLE comprises 2,088 multidisciplinary, open-ended questions that push models to the limits of their reasoning abilities, often revealing brittle generalisation and poor calibration under genuine uncertainty11. Finally, GPQA-Diamond38 features 198 graduate-level, multiple-choice questions across physics, chemistry, and biology. While its performance levels are comparable to those on Omni-MATH, its broader scientific scope provides a useful test of cross-domain generalisation under similarly saturated model conditions. Note that the GPQA benchmark was run at a different time point, i.e. October 2025 versus February 2025. For HLE and Omni-MATH, we prompt each model with a single, standardised template designed to elicit a clear explanation, a concise final answer, and a self-reported confidence score (0 %–100 %). We use a direct confidence elicitation format which closely follows the HLE benchmark11. This verbalized confidence elicitation approach17,43 captures models’ self-stated belief in their correctness, providing a simple, interpretable, and model-agnostic measure of certainty. Alternative confidence proxies, such as token-level probability-of-correctness estimates17, require logit access unavailable for closed-source models (e.g., Claude 3.7 Sonnet) and were thus not considered here. For GPQA-Diamond, we employed a slightly adapted version of the prompt provided by the Epoch AI benchmark suite44 to align with its multiple-choice format and encourage well calibrated responses17. Additionally, for Claude Sonnet 3.7, we set its max tokens to 128k and its thinking budget to 100k tokens. Similarly, for Qwen-253B-Think, we use the model presets of 131.1k max tokens, 32.2k of which can be used for output. DeepSeek-R1 has a max tokens parameter set to 128k, which includes the CoT tokens. To grade the final responses, we employ a customised grading strategy that accommodates the unique challenges and features of each benchmark. Omni-MATH’s responses were automatically scored using Omni-Judge37, an automatic mathematical grader, whose alignment with human judgment has been explicitly validated for Omni-MATH28,37,45. We prompted Omni-Judge as follows: For HLE, each answer from DeepSeek-R1 and Claude 3.7 Sonnet is independently assessed by both a human expert and a different LLM. We use OpenAI o3-mini to avoid preference leakage, which can occur when the same model is used for both answer generation and evaluation, potentially resulting in more lenient or biased assessments of its own responses46. The agreement between human and automated grades is high: 93.9% for Claude 3.7 Sonnet and 93.4% for DeepSeek-R1, corresponding to Cohen’s kappa values of 0.88 and 0.87, indicating near-perfect alignment. For further analysis, we include only HLE items where human and OpenAI o3-mini evaluations agree. For Qwen-235B-Think, grading was performed exclusively using OpenAI’s newer o4-mini, which provides improved calibration and evaluation robustness relative to the o3-mini-judge used on the answers of the other two reasoning models. Finally, to assess model calibration, we compute the expected calibration error (ECE), by grouping the outputs into confidence bins and calculate the mean absolute difference between the average reported confidence of each bin and its empirical precision. Across the three benchmarks, DeepSeek-R1, Claude Sonnet 3.7, and Qwen-235B-Think perform broadly in line with their official benchmark results, showing no single model as a consistent winner in accuracy. Note that a small derivation is to be expected as47 argue, point estimates on small, specialised tests should be reported with confidence intervals to avoid underestimating uncertainty (e.g., error bars or significance tests). On HLE, Qwen achieves the highest accuracy (13.5%), followed by Claude (9.2%) and DeepSeek-R1 (8.6%). On Omni-MATH, DeepSeek-R1 leads (72.5%) ahead of Claude (69.1%) and Qwen (60.6%), while on GPQA, Qwenn attains the highest accuracy (77.8%), closely followed by Claude (75.8%). On GPQA DeepSeek-R1 is a distant third scoring (61.6%), 9% below their official score. Calibration patterns, however, reveal a more systematic trend: all models exhibit strong confidence in their predictions, with most responses clustered near maximal self-reported certainty (Appendix B). Among them, Qwen-235B-Think achieves the lowest ECE across all benchmarks, however this pattern is not reflected in alternative measures of calibration such as the Macro Callibration Error (MacroCE)33,48,49 and Brier score50. A global overview of accuracies and calibration errors is presented in Table 1. A full breakdown, categorised by difficulty level (only for Omni-MATH), is provided in Appendix A.Table 1Overall accuracy and calibration error for DeepSeek-R1, Claude 3.7 Sonnet, and Qwen-235B-Think across the HLE, Omni-MATH, and GPQA benchmarks.DatasetModelAccuracy (%)ECE (%)MacroCE (%)Brier Score (%)Missing Answer (%)Missing Confidence Score (%)HLEDeepSeek-R1 (Jan)8.678.349.070.80.00.0Claude 3.7 Sonnet9.282.955.578.80.00.0Qwen-235B-Think13.578.057.774.920.020.0Omni-MATHDeepSeek-R1 (Jan)72.522.232.122.10.09.5Claude 3.7 Sonnet69.129.147.229.00.02.0Qwen-235B-Think60.68.845.810.131.636.8GPQADeepSeek-R1 (Jan)61.629.341.827.90.00.0Claude 3.7 Sonnet75.817.531.220.40.00.0Qwen-235B-Think77.816.140.818.91.01.0While Claude achieves the highest accuracy on GPQA, DeepSeek-R1 performs best on Omni-MATH, and Qwen leads on HLE. Calibration error remains highest for Claude across all benchmarks, suggesting overconfidence, while Qwen shows markedly better calibration on Omni-MATH but with a substantial number of missing confidence scores (1,631 omitted cases). Expected Calibration Error (ECE) is computed, as in51, by computing the weighted mean absolute difference between predicted confidence and empirical accuracy across binned confidence levels reported by the model. We also report Macro Calibration Error (MacroCE) or Mean Absolute Deviation33,48,49, the unweighted mean absolute calibration gap across non-empty bins, and the Brier score50, the mean squared error between predicted probabilities (self-reported confidence) and binary outcomes. Table 1Overall accuracy and calibration error for DeepSeek-R1, Claude 3.7 Sonnet, and Qwen-235B-Think across the HLE, Omni-MATH, and GPQA benchmarks.DatasetModelAccuracy (%)ECE (%)MacroCE (%)Brier Score (%)Missing Answer (%)Missing Confidence Score (%)HLEDeepSeek-R1 (Jan)8.678.349.070.80.00.0Claude 3.7 Sonnet9.282.955.578.80.00.0Qwen-235B-Think13.578.057.774.920.020.0Omni-MATHDeepSeek-R1 (Jan)72.522.232.122.10.09.5Claude 3.7 Sonnet69.129.147.229.00.02.0Qwen-235B-Think60.68.845.810.131.636.8GPQADeepSeek-R1 (Jan)61.629.341.827.90.00.0Claude 3.7 Sonnet75.817.531.220.40.00.0Qwen-235B-Think77.816.140.818.91.01.0While Claude achieves the highest accuracy on GPQA, DeepSeek-R1 performs best on Omni-MATH, and Qwen leads on HLE. Calibration error remains highest for Claude across all benchmarks, suggesting overconfidence, while Qwen shows markedly better calibration on Omni-MATH but with a substantial number of missing confidence scores (1,631 omitted cases). Expected Calibration Error (ECE) is computed, as in51, by computing the weighted mean absolute difference between predicted confidence and empirical accuracy across binned confidence levels reported by the model. We also report Macro Calibration Error (MacroCE) or Mean Absolute Deviation33,48,49, the unweighted mean absolute calibration gap across non-empty bins, and the Brier score50, the mean squared error between predicted probabilities (self-reported confidence) and binary outcomes. Overall accuracy and calibration error for DeepSeek-R1, Claude 3.7 Sonnet, and Qwen-235B-Think across the HLE, Omni-MATH, and GPQA benchmarks. While Claude achieves the highest accuracy on GPQA, DeepSeek-R1 performs best on Omni-MATH, and Qwen leads on HLE. Calibration error remains highest for Claude across all benchmarks, suggesting overconfidence, while Qwen shows markedly better calibration on Omni-MATH but with a substantial number of missing confidence scores (1,631 omitted cases). Expected Calibration Error (ECE) is computed, as in51, by computing the weighted mean absolute difference between predicted confidence and empirical accuracy across binned confidence levels reported by the model. We also report Macro Calibration Error (MacroCE) or Mean Absolute Deviation33,48,49, the unweighted mean absolute calibration gap across non-empty bins, and the Brier score50, the mean squared error between predicted probabilities (self-reported confidence) and binary outcomes. Intra-CoT sentiment volatility in the chain-of-thought The ability to regulate emotions, such as maintaining composure or overcoming moments of doubt, has been linked to improved cognitive outcomes in humans52–55. Motivated by this, we examine the impact of the Intra-CoT sentiment volatility within the CoT responses on the response correctness. Specifically, we asked whether the model’s reasoning began with a confident, optimistic tone and ended in doubt, or if it started hesitantly and grew more optimistic as the answer developed. To quantify this, we process each CoT through OpenAI o3-mini as an independent sentiment evaluator56. For each rationale, we extract two sentiment scores—the most negative and the most positive sentiment, ranging from -1 (strongly negative) to +1 (strongly positive). We also record the order in which these sentiments appear in the CoT, i.e., whether the response moved from negative to positive (an “upward” direction, scored as 1), from positive to negative (“downward,” scored as –1), or remained neutral throughout (scored as 0). We use the following prompt to evaluate each CoT: Using the sentiment scores provided by OpenAI o3-mini, we define Intra-CoT sentiment volatility as the signed difference between the most positive and most negative sentiment within each CoT. Large differences (Δ) may indicate moments of uncertainty, backtracking, or sudden shifts in confidence, while smaller deltas suggest a more consistent and steady tone throughout the reasoning process. Lexicographic analysis of the chain-of-thought To explore the connection between the linguistic characteristics of model-generated rationales and their accuracy, we perform a lexicographic analysis of CoT responses from all evaluated datasets and models. For each unique word found within a CoT, we calculate its relative accuracy by comparing the success rate of responses containing that word to the average accuracy within the respective dataset. To ensure reliability, we exclude low-frequency words by requiring that each word appear at least a minimum number of times (300 occurrences) in every dataset to be included in the analysis. Hedging words and accuracy In addition to the general lexicographic analysis, we focus on a specific subset of the lexicon: hedging words. In human communication, hedging—through modal verbs (e.g., might, could), uncertainty adverbs (e.g., possibly, perhaps), and tentative phrases (e.g., it seems that)—serves as a cue indicating lower confidence57–59. We hypothesise that the same linguistic uncertainty cues may also appear in the CoT of reasoning models. Prior work has shown that non-reasoning LLMs tend to underuse epistemic markers, often defaulting to overconfident phrasing even when incorrect60, motivating our investigation of whether hedging functions differently in explicit reasoning traces. Drawing on established taxonomies of uncertainty markers,57–59, we compiled a lexicon of such expressions and computed a hedging rate, i.e., proportion of sentences in a rationale containing at least one hedging expression CoT. The full hedging lexicon used was:Modal and uncertainty verbs: might, may, could, should, would, seems, suggests, appearsUncertainty adverbs: possibly, perhaps, likely, unlikely, probably, generally, usually, sometimes, often, tends, somewhat, rather, quite, almost, nearly, virtually, presumably, arguably, relatively, fairly, reasonably, mostly, partially, mainly, primarily, essentially, basicallyCommon hedging phrases: it seems that, it appears that, it suggests that, it is possible that, it is likely thatAdditional qualifiers: in part, to some extent Modal and uncertainty verbs: might, may, could, should, would, seems, suggests, appears Modal and uncertainty verbs: might, may, could, should, would, seems, suggests, appears Uncertainty adverbs: possibly, perhaps, likely, unlikely, probably, generally, usually, sometimes, often, tends, somewhat, rather, quite, almost, nearly, virtually, presumably, arguably, relatively, fairly, reasonably, mostly, partially, mainly, primarily, essentially, basically Uncertainty adverbs: possibly, perhaps, likely, unlikely, probably, generally, usually, sometimes, often, tends, somewhat, rather, quite, almost, nearly, virtually, presumably, arguably, relatively, fairly, reasonably, mostly, partially, mainly, primarily, essentially, basically Common hedging phrases: it seems that, it appears that, it suggests that, it is possible that, it is likely that Common hedging phrases: it seems that, it appears that, it suggests that, it is possible that, it is likely that Additional qualifiers: in part, to some extent Additional qualifiers: in part, to some extent Results Relationship between CoT length and accuracy To assess how the length of the CoT relates to response correctness, we analyze accuracy as a function of the number of words in each CoT rationale. We observe a downward trend in the Omni-MATH and GPQA benchmarks: accuracy declines as CoT length increases (see Fig. 1). Claude 3.7 Sonnet proves more robust to longer reasoning chains than DeepSeek-R1 and Qwen-235B, losing on average 3% of accuracy per additional 1,000 CoT words, compared to 4.4% for Qwen-235B and 5.8% for DeepSeek-R1. These results echo the findings of28, who report that o3-mini-high loses about 0.8 % accuracy per 1,000 additional reasoning tokens on Omni-MATH benchmark. A similar pattern extists for GPQA, where DeepSeek-R1 and Qwen-235B lose 11.1% and 8.4% accuracy per 1000 additional CoT words. We do not report this trend for Claude-3.7 Sonnet, as all of reasoning chains were uncharacteristically short, all less than 1000 words, for the GPQA benchmark. This could be a consequence of the later date on which GPQA was run. In contrast to the downward pattern reproted for Omni-MATH and GPQA, in HLE, we find that the CoT length shows no discernible pattern, accuracy stays low, and fluctuates widely regardless of CoT length. This suggests that CoT length predicts correctness only for benchmarks of intermediate difficulty—that is, tasks within the model’s demonstrated capabilities but not yet saturated. These findings are consistent with previous work showing a negative relationship between CoT length and response accuracy on Omni-MATH28, as well as the disappearance of this effect on the most challenging benchmarks30,61. This pattern may reflect self-doubt–driven overthinking when problems are within the model’s capability, whereas on harder benchmarks performance remains uniformly low regardless of reasoning length62. In general, we find that Claude Sonnet 3.7 generally produces longer CoTs than DeepSeek-R1 and Qwen-235B, which may be due to architectural choices or differences in reinforcement learning post-training3,6. In contrast, Claude Sonnet 3.7 produced the shortest CoT on GPQA, which may reflect the timing of the evaluation conducted in October rather than February, when the model was closer to its release.Fig. 1Accuracy as a function of CoT length (top) and the distribution of CoT lengths (bottom) across all benchmarks and model families. Across Omni-MATH and GPQA, accuracy decreases with increasing chain length for all three models. On HLE, accuracy remains uniformly low regardless of length. Claude consistently produces the longest and most variable reasoning chains, whereas Qwen tends to generate the shortest reasoning chains. CoT length bins with fewer than 30 samples were excluded from the accuracy plot. The GPQA–Claude results appear as a single point with fences, as Claude’s reasoning chains were uncharacteristically short for this benchmark. We speculate this might be due to a platform update on Anthropic’s side between benchmark runs. Accuracy as a function of CoT length (top) and the distribution of CoT lengths (bottom) across all benchmarks and model families. Across Omni-MATH and GPQA, accuracy decreases with increasing chain length for all three models. On HLE, accuracy remains uniformly low regardless of length. Claude consistently produces the longest and most variable reasoning chains, whereas Qwen tends to generate the shortest reasoning chains. CoT length bins with fewer than 30 samples were excluded from the accuracy plot. The GPQA–Claude results appear as a single point with fences, as Claude’s reasoning chains were uncharacteristically short for this benchmark. We speculate this might be due to a platform update on Anthropic’s side between benchmark runs. Intra-CoT sentiment volatility and reasoning dynamics Figure 2 shows the relationship between Intra-CoT sentiment volatility within CoT and response correctness. We observe that a predominantly parabolic pattern emerges for Omni-MATH, with accuracy peaking at a volatility value of approximately 0.1. This suggests that CoTs with a consistent or slightly uplifting sentiment are associated with the highest accuracy. The answers asociated with GPQA, tend to be more correct as a model’s exhibits neutral or less downward sloping reasoning. In contrast, no such pattern is observed for HLE, where accuracy remains uniformly low regardless of the degree of sentimental variation. To rule out the possibility that a nonzero mean (baseline) emotion score was driving this effect, we repeated the volatility?accuracy analysis on mean-centred sentiment values. After centring, we found the same parabolic, upward trend for Omni-MATH and GPQA, and the same flat pattern for HLE. While all three reasoning models display a predominantly neutral CoT, they exhibit markedly different sentimental footprints. Claude 3.7 Sonnet’s responses tend to be slightly optimistic in tone, becoming more positive over the course of their reasoning, while DeepSeek’s and Qwen rationales often exhibit a slight negative shift, see Fig. 2(right).Fig. 2Accuracy as a function of intra-CoT sentiment volatility (left) and its distribution (right) across all benchmarks and model families. On Omni-MATH and GPQA, accuracy peaks for CoTs with low or slightly positive volatility, suggesting that mild emotional fluctuation within the reasoning process supports better outcomes. In contrast, HLE shows uniformly lower accuracy across all volatility levels. Claude 3.7 Sonnet tends toward positive sentiment, while DeepSeek-R1 and Qwen-235B lean toward mildly negative reasoning chains. For GPQA–Claude, only a single averaged point with fences is shown due to the limited sample size at each volatility level. Bins with fewer than 30 samples were excluded. Accuracy as a function of intra-CoT sentiment volatility (left) and its distribution (right) across all benchmarks and model families. On Omni-MATH and GPQA, accuracy peaks for CoTs with low or slightly positive volatility, suggesting that mild emotional fluctuation within the reasoning process supports better outcomes. In contrast, HLE shows uniformly lower accuracy across all volatility levels. Claude 3.7 Sonnet tends toward positive sentiment, while DeepSeek-R1 and Qwen-235B lean toward mildly negative reasoning chains. For GPQA–Claude, only a single averaged point with fences is shown due to the limited sample size at each volatility level. Bins with fewer than 30 samples were excluded. Lexical hints of uncertainty To scrutinise how specific words reflect or undermine a model’s underlying confidence in response correctness, we construct a lemmatised lexicon from the CoT. We exclude rare or highly exclusive words from the lemmatisation process, retaining only those appearing in at least 300 CoT responses for each model and benchmark. We distinguish between lemmatised words that consistently harm or boost accuracy. For ease of reading, we use words to refer to the lemmas in the remainder of the paper. Interestingly, many of the words most strongly linked to reduced accuracy, such as complicated, complex, likely, probably, often, unless, are intuitively associated with task difficulty and cognitive overload, mirroring language that humans use to express uncertainty or task difficulty57–59,63. This suggests a degree of linguistic convergence in the signalling of uncertainty between reasoning models and humans. As shown in Fig. 3, the top 25 words most detrimental to accuracy also include terms conveying confusion or lack of direction, such as depend, unless, else, direction. A full overview of these lexical hints of uncertainty, i.e., “harmful” words can be found in Appendix 4. In contrast, only the word “equation” appears to have consistently boosted accuracy across all datasets (see Figure C). However, when considering its confidence interval, this apparent effect is likely due to random variation rather than a true underlying pattern. As a general rule, the words with the highest mean relative accuracy tend to be those characteristic of mathematical reasoning, including equivalent, solve, therefore, equal, equation, calculation, formula, compute, and coefficient. This is an artefact of Claude, DeepSeek, and Qwen demonstrating above-average performance on mathematical questions within HLE, achieving accuracies of 10%, 9.1%, and 13%, respectively.Fig. 3Relative accuracy of the 30 most detrimental lemmatised words across datasets and models. t Many of the most harmful words (e.g., complicated, complex, likely, probably, depend, often, unless) express uncertainty, cognitive overload, or confusion. The GPQA benchmark was excluded from this analysis because its limited size prevents the detection of robust lexical signals (i.e., words appearing \ge 300 times). When the occurrence threshold is lowered (e.g., n = 50), the benchmark becomes noisy and sparse (only outputting 45 words), with spurious signals driven by words that appear verbatim in the question text (e.g., group, structure, position, product).. Relative accuracy of the 30 most detrimental lemmatised words across datasets and models. t Many of the most harmful words (e.g., complicated, complex, likely, probably, depend, often, unless) express uncertainty, cognitive overload, or confusion. The GPQA benchmark was excluded from this analysis because its limited size prevents the detection of robust lexical signals (i.e., words appearing \ge 300 times). When the occurrence threshold is lowered (e.g., n = 50), the benchmark becomes noisy and sparse (only outputting 45 words), with spurious signals driven by words that appear verbatim in the question text (e.g., group, structure, position, product).. Hedging We continue by looking at a subset of our full lexicon, hedging words. Hedging words are typically used by humans to mark doubt or a lack of knowledge57–59; the same could be true for LLM-generated CoT. Across the eight of the nine model benchmark pairs, we observe a negative relationship between hedging rate and accuracy: as the proportion of hedged sentences increases, the accuracy tends to decline (Fig. 4). This trend is most pronounced for Omni-MATH, where the correlations are r=-0.24 (p<0.001) for DeepSeek-R1 and r=-0.14 (p<0.001) for Claude 3.7 Sonnet, indicating small but statistically detectable negative associations under the null hypothesis of zero correlation. In contrast, Qwen-235B-Think exhibits a weak positive correlation (r=0.06, p<0.001), a possible effect from the large amount of unanswered questions. For HLE, the correlations are weaker: r=-0.10 (p<0.001) for DeepSeek-R1, r=-0.04 (p=0.06) for Claude 3.7 Sonnet, and r=-0.07 (p<0.01) for Qwen-235B-Think. All three exhibit small negative effects, though only DeepSeek and Qwen reach statistical significance. The hedging relationship is the most pronounced for GPQA, where correlations reach r=-0.38 (p<0.001) for DeepSeek-R1, r=-0.21 (p<0.01) for Claude 3.7 Sonnet, and r=-0.25 (p<0.001) for Qwen-235B. Across all benchmarks, higher hedging rates are generally linked to lower accuracy—most clearly in Omni-MATH and GPQA—though the effect weakens on HLE. Median hedging rates were about 9% on Omni-MATH, 18% on HLE, and 22% on GPQA.Fig. 4Accuracy as a function of hedging rate (left) and the distribution of hedging rates (right) for each model and benchmark. Higher hedging rates are generally associated with lower accuracy, particularly on Omni-MATH and GPQA. The effect is weaker for HLE and nearly absent for Claude 3.7 Sonnet. Hedging is most common in GPQA (median \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 22\%$$\end{document}) and HLE (median \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 18\%$$\end{document}), while low for Omni-MATH (median \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 9\%$$\end{document}). As before, bins with fewer than 30 samples were excluded from the left figure. Accuracy as a function of hedging rate (left) and the distribution of hedging rates (right) for each model and benchmark. Higher hedging rates are generally associated with lower accuracy, particularly on Omni-MATH and GPQA. The effect is weaker for HLE and nearly absent for Claude 3.7 Sonnet. Hedging is most common in GPQA (median \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 22\%$$\end{document}) and HLE (median \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 18\%$$\end{document}), while low for Omni-MATH (median \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 9\%$$\end{document}). As before, bins with fewer than 30 samples were excluded from the left figure. Prediction accuracy The question now remains whether these lexicographic features can be used to predict/anticipate the correctness of a reasoning model’s final response. To assess which features best predict the correctness of model-generated reasoning, we train a neural network on both general features (CoT length, sentiment volatility, and hedging rate) and with the 25 most consistently “harmful” non-lemmatised words (see Appendix 3, and combinations between the standard features and “harmful” words. To prevent lexical leakage, the harmful-word lexicon is rebuilt for each of the 30 random seeds using only the training data. Each model is trained on one benchmark and evaluated both within and across benchmarks to test in- and cross-benchmark predictive capability. The GPQA benchmark is used exclusively as a held-out test set, given its smaller size, which prevents consistent identification of lexical cues, (e.g., words appearing at least 300 times), which is especially true for the uncharacteristically short reasoning chains of Claude Sonnet 3.7 on GPQA. For all experiments, we employ a feed-forward neural network with two hidden layers (32 and 16 units, ReLU activations) and a sigmoid output. Models are trained using the Adam optimiser and binary cross-entropy loss. Given the strong class imbalance in the training data, particularly for HLE, whose accuracy for the minority class was only 8.9%, it is crucial to prevent the model from exploiting this imbalance with trivial predictions (e.g., always guessing “incorrect” for DeepSeek’s answers on HLE would yield an accuracy of 91.4%). Consequently, we apply class weighting during neural network training, penalising mistakes on the minority class more heavily, to encourage the model to learn non-trivial, meaningful patterns for both classes. All experiments are conducted on 30 seeds (0-29) to ensure robust estimation, with the results averaged across multiple runs. Training is performed on 80% of the original unbalanced data; we then balance the remaining 20% so that evaluation sets contain the same number of correct and incorrect examples, ensuring that performance consistently above 50% is indeed better than random guessing. Figure 5 shows that lexical features alone are the strongest individual predictors of correctness, generalising well both within and across benchmarks. Adding additional features, such as CoT length, hedging rate, or sentiment dynamics, does not improve predictive accuracy and, in some cases, even decreases it. The only exception is the combination of lexical features with self-reported confidence, which yields the highest mean accuracy across all experimental settings. Notably, CoT length, which has been highlighted in previous studies28–32, consistently provides the least predictive value and harms performance when included as an additional feature in all of our analyses. The ROC curves for each analysis can be found in Appendix D. To assess whether the lexical signal merely captures domain composition (i.e., non math-related terminology) or interacts with reasoning length, we conducted an auxiliary domain-sensitivity analysis that split the HLE test set into mathematical and non-mathematical questions while jointly controlling for CoT length, see Table 5. The lexical-only models performed worse on math than on non-math items (0.52 vs. 0.56 accuracy). When reasoning length was added as a covariate, this difference disappeared—but overall accuracy declined—indicating that CoT length partly mediates domain variance without providing additional discriminative power. Adding confidence to the lexical markers also levels performances over mathematical and non-mathematical questions, but at the same time improves accuracy to aproximately 0.58. These results support the view that CoT length acts as a coarse marker of reasoning effort that is most informative for intermediate-difficulty regimes (e.g., Omni-MATH, GPQA). Furthermore, they reflect what we find in the neural networks, who show that the models trained on the reasoning length of Omni-Math perform well within omni-math and GPQA but do not generalize to the harder HLE. Overall our findings suggest that a small set of words offers an important and potentially generalizable signal for predicting final response correctness, serving as lexical hints of uncertainty within CoT reasoning. The complete confusion matrices and Matthews Correlation Coefficient (MCC) for the best-performing neural network, which was trained using only these 25 words, are available in Appendix Table 6.Fig. 5Each group of bars represents a different combination of input features: a bag-of-words model using 25 “harmful” words (“Words”), common surface features (reasoning length, sentiment, hedging), or their combinations. Neural networks were trained on imbalanced data but evaluated on balanced test sets to ensure that accuracy reflects genuine generalisation rather than label bias. Results are averaged across 30 random seeds, with error bars showing variability across runs. For each seed, the harmful-word lexicon was recompiled exclusively from the training data, preventing any information leakage. In-benchmark performance (blue: HLE → HLE, orange: Omni → Omni) remains consistently higher than cross-benchmark generalisation (green: HLE → Omni, red: Omni → HLE, purple: HLE → GPQA, brown: Omni → GPQA). The GPQA benchmark was used exclusively as a test set due to its smaller size, which prevents consistent identification of lexical cues (e.g., words occurring at least 300 times). Adding sentiment or hedging features provided limited benefit, while incorporating reasoning length generally reduced both in- and cross-benchmark accuracy. Overall, a compact, seed-specific lexicon of 25 harmful words proved to be the most robust and generalizable predictor of answer correctness across all three benchmarks.. Each group of bars represents a different combination of input features: a bag-of-words model using 25 “harmful” words (“Words”), common surface features (reasoning length, sentiment, hedging), or their combinations. Neural networks were trained on imbalanced data but evaluated on balanced test sets to ensure that accuracy reflects genuine generalisation rather than label bias. Results are averaged across 30 random seeds, with error bars showing variability across runs. For each seed, the harmful-word lexicon was recompiled exclusively from the training data, preventing any information leakage. In-benchmark performance (blue: HLE → HLE, orange: Omni → Omni) remains consistently higher than cross-benchmark generalisation (green: HLE → Omni, red: Omni → HLE, purple: HLE → GPQA, brown: Omni → GPQA). The GPQA benchmark was used exclusively as a test set due to its smaller size, which prevents consistent identification of lexical cues (e.g., words occurring at least 300 times). Adding sentiment or hedging features provided limited benefit, while incorporating reasoning length generally reduced both in- and cross-benchmark accuracy. Overall, a compact, seed-specific lexicon of 25 harmful words proved to be the most robust and generalizable predictor of answer correctness across all three benchmarks.. Discussion and conclusion Our results reveal a clear information hierarchy within CoT traces. Lexical uncertainty hints dominate: tokens such as complicated, probably, and likely reduce accuracy odds by up to 25 % relative to baseline (Fig.3). Intra-CoT sentiment volatility is weaker but complementary; a modest upward sentiment shift (Δ =0.1) or upward slope in emotion coincide with improved accuracy on Omni-MATH and GPQA, yet sentiment is uninformative on HLE (Fig.2). On the moderate difficulty benchmarks Omni MATH and GPQA we find a negative correlation between CoT length and answer accuracy, whereas no such link appears on the more demanding HLE set. This pattern matches earlier work reporting accuracy declines with longer CoTs on Omni MATH28 and the absence of this effect on highly challenging benchmarks30,61. This asymmetry can be operationalised into a simple yet effective heuristic: filter out responses whose CoT contains strong uncertainty words, regardless of their length or self-reported confidence. Even a small lexicon of such “harmful” words performs on par with, or better than, more elaborate confidence-based approaches. Specifically, training a binary classifier using only the 25 most consistently harmful non-lemmatised words achieves Matthews correlation coefficients (MCCs) of 0.151 when trained and tested on HLE (and 0.158 cross-evaluated on Omni-MATH), 0.219 when trained and tested on Omni-MATH (0.113 cross-evaluated on HLE), and up to 0.278 and 0.282 when evaluated on GPQA. In contrast, simple confidence-thresholding heuristics yield substantially lower correlations, underscoring that certain lexical cues alone could serve as robust indicators of reasoning accuracy. Strikingly, a lightweight rule, i.e. mark an answer wrong if any of the top 5 harmful words appear in its CoT, otherwise classify as correct, achieves MCC = 0.144 on HLE, 0.322 on Omni-Math, and 0.390 on GPQA, see Appendix Table 7. This suggests that a handful of carefully selected lexical cues could serve as a highly practical post?hoc filter. A further advantage of this technique is that it treats the CoT as plain text: no weight access, no second inference pass is necessary. The contrast between DeepSeek-R1 and Claude 3.7 Sonnet might illuminate the role of RLHF stylistics. Claude tends to generate longer, optimism-biased CoTs yet shares the same lexical uncertainty fingerprint as R1. This suggests that affective style might be an alignment artefact, whereas the full chain may not mirror the true multi-step search in latent space; the presence of hedging words does correlate with lower internal confidence, which is in line with23. Limitations and future directions In this paper, two English benchmarks were studied, so our findings might not hold up in languages with richer morphology and different hedging forms. Furthermore, our hedging list is fixed and English-focused, meaning we miss any new or non-English uncertainty phrases. The sentiment model we used (OpenAI o3-mini) brings its own calibration quirks. Using multiple raters or a human-scored baseline could smooth out existing noise. Additionally, our HLE grading depends on LLM–human agreement, which implies that the mislabels could hide subtle effects. A related limitation concerns Qwen-235B-Think, whose responses were evaluated exclusively by OpenAI’s o4-mini model. While o4-mini offers improved calibration and grading robustness relative to o3-mini, the absence of human cross-validation means that potential judgment errors by the evaluator cannot be fully ruled out. Finally, correlation is not causation: those uncertainty markers could simply be more common around tougher, hidden problems in general. Key avenues for future work include: (i) multilingual replication with adaptive hedging lexicons; (ii) adversarial prompting to force models to suppress hedging and test the causal impact on accuracy; (iii) prompt-sensitivity and “thinking-budget” analyses that vary CoT styles or prompt framing to assess the robustness of lexical and reasoning-length effects; (iv) examining how lexical uncertainty markers relate to item-level question hardness; and (v) to understand RLHF’s impact on uncertainty expression, future work could track changes in hedging frequency and harmful token usage between SFT and RLHF checkpoints. This includes testing whether RLHF amplifies or suppresses the lexical signals that predict incorrect reasoning. Lastly, it is known that LLMs are often fine-tuned to avoid producing certain “harmful” words; in practice this may block or distort a reasoning chain that could otherwise have led to a truthful answer. Limitations and future directions In this paper, two English benchmarks were studied, so our findings might not hold up in languages with richer morphology and different hedging forms. Furthermore, our hedging list is fixed and English-focused, meaning we may miss relevant or non-English uncertainty phrases. Our measurement pipeline introduces additional noise. Sentiment is evaluated using a single model (OpenAI o3-mini), and HLE grading relies on LLM–human agreement, which may mask subtle effects. A related limitation concerns Qwen-235B-Think, whose responses were evaluated exclusively by OpenAI’s o4-mini model. While o4-mini offers improved calibration and grading robustness relative to o3-mini, the absence of human cross-validation means that potential judgment errors by the evaluator cannot be fully ruled out. Finally, all reported relationships are correlational. The association between lexical uncertainty markers and lower accuracy may reflect underlying task difficulty rather than intrinsic model uncertainty, i.e., harder problems may induce both more hedging and higher error rates. The proposed signals should therefore be interpreted as predictive rather than causal. Key avenues for future work include: (i) multilingual replication with adaptive hedging lexicons; (ii) adversarial prompting to force models to suppress hedging and test the causal impact on accuracy; (iii) prompt-sensitivity and “thinking-budget” analyses that vary CoT styles or prompt framing to assess the robustness of lexical and reasoning-length effects; (iv) examining how lexical uncertainty markers relate to item-level question hardness; and (v) to understand RLHF’s impact on uncertainty expression, future work could track changes in hedging frequency and harmful token usage between SFT and RLHF checkpoints. This includes testing whether RLHF amplifies or suppresses the lexical signals that predict incorrect reasoning. Lastly, as LLMs are often fine-tuned to avoid producing certain “harmful” words, this may suppress or distort reasoning traces without improving underlying correctness. Broader impact statement This work shows that simple lexical cues in an LLM’s Chain-of-Thought can help predict likely errors, offering a lightweight and model-agnostic way to improve calibration. Such signals could support safer deployment of reasoning models in high-stakes domains. Risks include adversarial evasion of these cues and false positives. It should be noted that these lexical cues could trigger Goodhart’s law: if they become widely used for filtering, model providers or users may fine-tune systems to suppress the flagged words without improving the underlying reasoning. Such stylistic suppression would undermine the predictive value of our signals while leaving true error rates unchanged. Mitigation of these risks may involve combining our method with other calibration techniques and extending tests to multilingual and domain-diverse settings. Broader testing across languages and domains is needed to ensure robustness, prevent misuse, and identify effects that may differ outside the English reasoning benchmarks studied.