Optimizing Chain-of-Thought Confidence via Topological and Dirichlet Risk Analysis
Abhishek More, Anthony ZhangNicole BonillaAshvik Vivekan Kevin ZhuParham Sharafoleslami, Maheep Chaudhary Algoverse AI Research{abhisheknmore, maheepchaudhary.research}@gmail.com, [0.3em] Equal contribution · 2025
Read Optimizing Chain-of-Thought Confidence via Topological and Dirichlet Risk Analysis by Abhishek More, Anthony ZhangNicole BonillaAshvik Vivekan Kevin ZhuParham Sharafoleslami, Maheep Chaudhary Algoverse AI Research{abhisheknmore, maheepchaudhary.research}@gmail.com, [0.3em] Equal contribution (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06437.txt.
Introduction
Abhishek More and six co-authors measure how much a language model's chain of thought can be trusted. Step-by-step prompting solves hard problems, but confidence scores from current methods are poorly calibrated and often highest on wrong answers. Their method, EDTR, scores confidence from the shape of many sampled reasoning paths.
Each reasoning chain becomes a vector, and eight topological features describe how the paths cluster. Tight, coherent clusters signal confidence, while scattered paths signal doubt, and a Dirichlet model turns that geometry into a risk estimate. Across math and commonsense benchmarks, calibration improved by about forty-one percent over leading rivals, with perfect accuracy on the olympiad-level test set.
Catalog record
| CREATOR | Abhishek More, Anthony ZhangNicole BonillaAshvik Vivekan Kevin ZhuParham Sharafoleslami, Maheep Chaudhary Algoverse AI Research{abhisheknmore, maheepchaudhary.research}@gmail.com, [0.3em] Equal contribution |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06437
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Chain-of-Thought Confidence via Topological and Dirichlet Risk Analysis Abhishek More, Anthony ZhangNicole BonillaAshvik Vivekan Kevin ZhuParham Sharafoleslami, Maheep Chaudhary Algoverse AI Research{abhisheknmore, maheepchaudhary.research}@gmail.com, [0.3em] Equal contribution Abstract Chain-of-thought (CoT) prompting enables Large Language Models to solve complex problems, but deploying these models safely requires reliable confidence estimates, a capability where existing methods suffer from poor calibration and severe overconfidence on incorrect predictions. We propose Enhanced Dirichlet and Topology Risk (EDTR), a novel decoding strategy that combines topological analysis with Dirichlet-based uncertainty quantification to measure LLM confidence across multiple reasoning paths. EDTR treats each CoT as a vector in high-dimensional space and extracts eight topological risk features capturing the geometric structure of reasoning distributions: tighter, more coherent clusters indicate higher confidence while dispersed, inconsistent paths signal uncertainty. We evaluate EDTR against three state-of-the-art calibration methods across four diverse reasoning benchmarks spanning olympiad-level mathematics (AIME), grade school math (GSM8K), commonsense reasoning, and stock price prediction 21 ; 3 ; 15 ; 20 . EDTR achieves 41% better calibration than competing methods with an average ECE of 0.287 and the best overall composite score of 0.672, while notably achieving perfect accuracy on AIME and exceptional calibration on GSM8K with an ECE of 0.107, domains where baselines exhibit severe overconfidence. Our work provides a geometric framework for understanding and quantifying uncertainty in multi-step LLM reasoning, enabling more reliable deployment where calibrated confidence estimates are essential. Figure 1: A sample prompt is put into an LLM and then enters an encoder model: all-MiniLM-L6-v2. Topology and Dirichlet Head combine to increase and produce highest confidence output. 1 Introduction Large Language Models (LLMs) are deployed across domains from software development to financial services, yet their widespread adoption exposes a critical vulnerability: users often trust LLM outputs without understanding the model’s uncertainty. While substantial research improves task accuracy, confidence calibration, ensuring a model’s expressed confidence matches its actual correctness, remains understudied. Recent work reveals fundamental limitations: CoT exhibits domain sensitivity 10 ; 14 and fails to capture the structural properties of reasoning paths 22 . Existing confidence estimation approaches, verbalized confidence 18 , self-consistency voting 10 , and token probability analysis suffer from poor calibration and frequent overconfidence on incorrect predictions. Recent calibration methods attempt to address these challenges but face critical limitations. We propose Enhanced Dirichlet+Topology Risk (EDTR), an inference-time framework that estimates LLM confidence by analyzing the topological properties of multiple reasoning paths in semantic vector space. Our key insight is geometric: the distribution of reasoning embeddings reveals structural properties of model uncertainty. EDTR generates diverse CoT reasoning paths, encoding them into high-dimensional semantic space, and extracting eight topological risk features quantifying geometric properties of the reasoning distribution. These features combine with Dirichlet-based uncertainty quantification to produce calibrated confidence estimates. Our contributions are: 1. A topological framework for analyzing LLM reasoning confidence through geometric properties of CoT embeddings. 2. Eight interpretable risk features capturing reasoning consistency, coherence, and cluster quality that enable explainable confidence estimates. 3. Comprehensive evaluation showing EDTR achieves superior performance with composite score of 0.672 and 41% better calibration than competing methods, achieving an average ECE of 0.287. Our work demonstrates that geometric analysis of reasoning distributions provides robust, generalizable confidence estimates across diverse domains, enabling safer LLM deployment in high-stakes applications. 2 Related Work Prior work on LLM confidence estimation includes task-specific approaches for sarcasm detection and stock prediction 11 ; 7 , and methods analyzing model responses alongside reasoning 16 . Recent calibration techniques like GrACE (Zhang, Liu, and Patras 2025) require costly fine-tuning, while our work combines Dirichlet-based uncertainty quantification with topological analysis to provide training-free confidence estimation. 3 Methodology 3.1 Problem Formulation Given a query q and a language model \mathcal{M} , we aim to estimate the model’s confidence in its prediction by analyzing the geometric structure of multiple reasoning paths. Let \{\text{o}_{1},\ldots,\text{o}_{k}\} denote k chain-of-thought trajectories sampled from \mathcal{M} for query q . Our goal is to produce a calibrated confidence score C\in[0,1] such that the model’s expressed confidence aligns with its empirical accuracy. 3.2 Framework Overview EDTR operates in three stages as shown in Figure 1 with following steps: (1) diverse CoT generation via temperature-varied sampling, (2) topological feature extraction from reasoning embeddings, and (3) fusion of topology-based risk with Dirichlet uncertainty quantification. For each query q , we generate k=5 conventional diverse reasoning paths from Llama-3.1-8B with LoRA adapters by sampling \mathcal{M} with varying temperature parameters \tau\in\{0.7,0.8,0.9,1.0,1.1\} to encourage exploration of the reasoning space while maintaining coherence 4 . Each generated \text{CoT}_{i} consists of intermediate reasoning steps followed by a final answer a_{i} . 3.3 Geometric Feature Extraction from Reasoning Embeddings We embed each chain-of-thought into a semantic space using sentence embeddings via the all-MiniLM-L6-v2 model, producing vectors \{\mathbf{e}_{1},\ldots,\mathbf{e}_{k}\}\subset\mathbb{R}^{384} 13 ; 19 . From this point cloud, we extract eight geometric risk features that quantify reasoning consistency and coherence: reasoning spread ( \sigma_{\text{dist}} ), consistency score ( C_{\text{cos}} ), complexity entropy ( E_{\text{comp}} ), stability score ( S_{\text{DBSCAN}} ), coherence score ( C_{\text{centroid}} ), diversity penalty ( P_{\text{div}} ), outlier risk ( R_{\text{outlier}} ), and cluster quality ( Q_{\text{sil}} ). These features collectively form a topological risk profile that captures both local coherence and global structure in the reasoning distribution, and are combined into a weighted aggregate risk score: \begin{split}\text{risk}_{\text{topo}}=&\,w_{1}\sigma_{\text{dist}}+w_{2}C_{\text{cos}}+w_{3}E_{\text{comp}}+w_{4}S_{\text{DBSCAN}}\\ &+w_{5}C_{\text{centroid}}+w_{6}P_{\text{div}}+w_{7}R_{\text{outlier}}+w_{8}Q_{\text{sil}}\end{split} (1) where \{w_{i}\}_{i=1}^{8} are learned weights. The numerical values for each of the learned weights are listed in the Appendix 8 . This topological risk profile captures both local coherence and global structure in the reasoning distribution, and is used in Section 3.5 to compute the final confidence score. 3.4 Dirichlet-Based Uncertainty Quantification In parallel, we quantify uncertainty using a learned Dirichlet-based approach that captures second-order uncertainty over predicted class probabilities. For each CoT trajectory i\in\{1,\ldots,k\} , we compute variance \sigma^{2}_{i} and entropy H_{i} of token-level probability distributions. These statistics are fed into a compact two-layer neural network (hidden dimensions 128 and 64) that predicts Dirichlet parameters \boldsymbol{\alpha}=(\alpha_{1},\ldots,\alpha_{n}) : \boldsymbol{\alpha}=\text{softplus}(\text{MLP}([\sigma^{2}_{1},H_{1},\ldots,\sigma^{2}_{k},H_{k}];\theta))+1 (2) The softplus transformation ensures \alpha_{i}>1 , yielding a proper Dirichlet distribution \text{Dir}(\boldsymbol{\alpha}) that models uncertainty about class probabilities themselves. The concentration parameter \alpha_{0}=\sum_{i=1}^{n}\alpha_{i} indicates epistemic confidence: higher values reflect certainty, while lower values signal ambiguity. We extract a four-dimensional Dirichlet feature vector \mathbf{f}_{\text{dir}}\in\mathbb{R}^{4} capturing: (1) concentration \alpha_{0} , (2) differential entropy H[\text{Dir}(\boldsymbol{\alpha})] , (3) expected maximum probability \max_{i}\alpha_{i}/\alpha_{0} , and (4) variance of the most probable class. To obtain a scalar confidence score, we compute: \text{entropy\_conf}=1\div\left(1+\sum_{i=1}^{n}\left[\psi(\alpha_{i})-\psi(\alpha_{0})\right]\right)\\ (3) \text{conf}_{\text{dir}}=1\div 3\left(\max_{i}\frac{\alpha_{i}}{\alpha_{0}}+\sigma(\alpha_{0}-n)+\text{entropy\_conf}\right) (4) where \sigma(\cdot) is the sigmoid function and \psi(\cdot) is the digamma function. This composite score balances expected probability, precision relative to the number of classes, and distributional sharpness. The confidence is clipped to [0.01,0.99] for numerical stability. 3.5 Confidence Fusion We combine topological and Dirichlet confidence scores using a calibrated fusion strategy as shown in Equation 5 . C=\sigma(w_{\text{topo}}\cdot\text{risk}_{\text{topo}}+w_{\text{dir}}\cdot\text{conf}_{\text{dir}}+b) (5) where \sigma(\cdot) is the sigmoid function with 60% weight to topological features and 40% to Dirichlet features. 4 Experimentation and Results 4.1 Datasets and Tasks We evaluate EDTR across four diverse reasoning benchmarks to assess calibration quality across different reasoning modalities. Our datasets include AIME (olympiad-level mathematics), GSM8K (grade school math word problems), CommonsenseQA (multiple-choice commonsense reasoning), and stock price prediction using S&P 500 data from Yahoo Finance. Each dataset was collected, cleaned,