Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
Xiaoou Liu, Tiejin Chen, Dengjia Zhang, Yaqing Wang, Lu Cheng, Hua Wei · 2026
Read Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution by Xiaoou Liu, Tiejin Chen, Dengjia Zhang, Yaqing Wang, Lu Cheng, Hua Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19228.txt.
Introduction
Xiaoou Liu and five colleagues introduce Stepwise Confidence Attribution, scoring each reasoning step for closed source models. Existing methods cover only final answers or need internal access. SCA uses the Information Bottleneck idea: steps matching consensus patterns across correct solutions score high, while deviating steps are flagged as wrong. Two methods are offered: NIBS, measuring consistency without graphs, and GIBS, learning subgraphs through a mask. Tests on math and multihop questions show SCA finds low confidence steps tied to errors, and step guided correction lifts success by 13.5 percent.
Catalog record
| CREATOR | Xiaoou Liu, Tiejin Chen, Dengjia Zhang, Yaqing Wang, Lu Cheng, Hua Wei |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73175 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19228
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution Xiaoou Liu, Tiejin Chen, Dengjia Zhang, Yaqing Wang, Lu Cheng, Hua Wei Abstract Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult. Confidence estimation offers a diagnostic signal, yet existing methods are restricted to final answers or require internal model access. We introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5% over answer-level feedback. 1 Introduction The ability to diagnose where a multi-step reasoning trace fails is essential for improving the reliability of large language models (LLMs). Solution traces such as Chain-of-Thought (CoT) ( 42 ) or Graph-of-Thought (GoT) ( 4 ) provide transparency into model reasoning, yet intermediate errors remain hard to identify and can critically affect the final prediction. Recent work has explored step-level diagnostics for reasoning traces, largely falling into two categories. The first trains supervised classifiers with step-by-step human annotations to label whether each reasoning step is correct ( 22 ; 48 ; 26 ) . The second prompts the LLM itself as a judge to critique each solution step by step ( 43 ; 25 ) . While both directions can provide useful signals, the former requires expensive human annotation, and the latter inherits bias and inconsistency from the judge model, limiting scalability and reliability. Figure 1 : Example of reasoning trace variability in GSM8K dataset. Two distinct solution paths (B and C) yield the same correct answer, while another path (A) contains an erroneous step leading to a wrong result. Stepwise confidence attribution needs to distinguish legitimate variability from true logical inconsistencies. Confidence estimation (CE) provides a complementary direction for assessing reliability, as it can operate directly on model outputs. Prior work has shown that measures such as semantic variance across sampled generations ( 27 ; 15 ) or predictive entropy over logits ( 28 ; 24 ) provide informative signals for estimating whether a final answer is correct. However, restricting CE to the final answer yields only a coarse reliability signal and fails to indicate which specific step in a reasoning trace is responsible for an error. This limitation motivates the need for Stepwise Confidence Attribution (SCA), where the goal is to assign confidence scores to individual reasoning steps and thereby provide fine-grained diagnostic signals. Extending confidence attribution from the final-answer level to the step-wise setting introduces a key challenge. Reasoning traces generated by LLMs exhibit substantial output variability : correct solutions may differ in step order, expression, or level of detail. Yet, despite these surface-level differences, correct reasoning paths are not random; they are governed by the underlying logic of the problem. Consequently, valid traces tend to converge on specific key logical invariants , i.e, critical intermediate states or semantic milestones that are necessary to derive the correct answer (e.g., calculating intermediate costs in a math problem). A robust SCA must distinguish this legitimate variability from true errors. While legitimate variability represents alternative trajectories between these logical anchors, errors act as deviations that drift away from the consensus of valid reasoning. For example, in Figure 1 , Solutions B and C arrive at the correct answer through different computation orders. Despite the permutation of steps, both solutions explicitly resolve the necessary intermediate values (the cost of pencils and notebooks). They share functional equivalence in their reasoning logic and should receive high confidence. In contrast, solution A, while structurally similar to B, introduces an incorrect operation at step 4; this deviation undermines the common structure and should receive low confidence. Our key idea is to aggregate multiple solution traces across correct solutions and identify logical invariants. These logical invariants serve as anchors of reliable reasoning and are assigned high confidence, while steps absent from consensus patterns are assigned low confidence, since they are more likely to reflect spurious or error-prone reasoning. Consensus patterns thus act as a proxy for the latent logical pattern underlying a problem, enabling fine-grained and robust confidence attribution. This intuition of finding a shared pattern amidst noisy variations maps naturally onto the Information Bottleneck (IB) principle. IB provides a formal language for balancing two competing objectives: compressing the input trajectory by discarding non-essential variations (the compression term), while retaining maximal information about the underlying correct reasoning pattern (the relevance term). Here, the input X is a reasoning trajectory composed of multiple steps, the compressed representation Z is the trajectory expressed with confidence weights on its steps, and the target Y denotes the correctness signal of the trajectory. The objective is \min_{Z}\;I(X;Z)-\beta I(Z;Y), where the first term encourages compression by selecting only a sparse subset of steps, and the second term ensures that this subset retains critical information required for correctness. Within this framework, we explore two complementary instantiations: • Non-parametric IB for Stepwise Confidence. Z is realized as a set of consensus steps derived from correct solutions, and step-level confidence is obtained by measuring how well a trajectory aligns with this set. • Graph IB for Stepwise Confidence (GIBS). To better handle structural variability, trajectories are represented as graphs, and Z is a subgraph selected through a differentiable mask. Confidence scores are produced by aligning the selected subgraph with correctness signals, providing a more flexible treatment. Empirical results show that both IB-based methods produce accurate confidence estimates. Beyond diagnostics, we demonstrate the impact of fine-grained CE on downstream LLM performance, showing that using step-level signals guides self-correction and improves correction success rate by up to 13.5% over answer-level feedback. Ablation studies verify the contribution of each component, and robustness analyses demonstrate label availability conditions, generalization across reasoning formats, and domain shift. 2 Related Work Confidence Estimation in LLMs. Confidence estimation (CE) aims at assigning reliability scores to individual outputs ( 27 ) . Most CE methods focus on the final answer, either through internal signals such as entropy ( 24 ; 28 ) , or through black-box signals such as agreement across sampled outputs ( 27 ) . However, these methods give no insight into intermediate steps. Recent stepwise CE approaches ( 45 ; 18 ) attempt to assign confidence along reasoning traces, but require internal access to token probabilities, restricting them to open-source models. Some works also model reasoning as graphs ( 4 ; 32 ) , but the representation itself is orthogonal to our problem setting. Our work differs by introducing the problem of stepwise confidence attribution in the black-box setting, which provides scalable confidence signals at the step level using only generated reasoning traces. Reasoning Verification in LLMs. Reasoning verification