Reverse Probing: Supervised Token-level Uncertainty Quantification for Large Language Models in Clinical Text
Bushi Xiao, Sarvesh Soni, Daisy Zhe Wang · 2026
Read Reverse Probing: Supervised Token-level Uncertainty Quantification for Large Language Models in Clinical Text by Bushi Xiao, Sarvesh Soni, Daisy Zhe Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28740.txt.
Introduction
Bushi Xiao, Sarvesh Soni, and Daisy Zhe Wang propose Reverse Probing, the first uncertainty framework specialized for clinical summarization. Most uncertainty methods target open-domain generation and cannot pin uncertainty to tokens or spans in long clinical text. Reverse Probing instead treats text as a window into the model's internal state, drawing uncertainty signals from four kinds of internal activations. On two expert-annotated clinical datasets it beats eight adapted baselines on all metrics, with up to four times higher AUPRC and lower inference time and compute costs.
Catalog record
| CREATOR | Bushi Xiao, Sarvesh Soni, Daisy Zhe Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53453 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28740
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reverse Probing: Supervised Token-level Uncertainty Quantification for Large Language Models in Clinical Text Bushi Xiao, Sarvesh Soni, Daisy Zhe Wang Abstract As large language models are increasingly deployed for clinical text, ensuring they can reliably signal their own uncertainty becomes critical. Most existing uncertainty quantification (UQ) methods are designed for open-domain generation and cannot localize uncertainty at the token or span level in long clinical text. We propose Reverse Probing, the first UQ framework specialized for clinical summarization, which estimates token-level uncertainty directly from pre-existing labeled summaries. Rather than sampling new outputs, Reverse Probing treats the text as a probe into the model’s internal state, extracting uncertainty signals from four categories of internal activations. We evaluate on two expert-annotated clinical datasets and outperform eight adapted baselines on all metrics, achieving up to 4 × 4\times higher AUPRC while reducing inference time and computational costs. Feature analysis reveals that delta energy and neighborhood context are the most consistent predictors across all models. This study offers interpretable insights into how models internally respond to unsupported clinical content. 1 Introduction Uncertainty quantification (UQ) for Large Language Models (LLMs) has been widely studied in general domains, yet its application to clinical text remains limited. Clinical text presents distinct challenges from general tasks: it is dense with terminology, errors often occur at a single word or number within an otherwise correct sentence, and small mistakes can carry serious consequences. Most existing UQ methods operate at the sequence level and are poorly suited for fine-grained, token-level uncertainty in long clinical text. UQ is also fundamentally distinct from hallucination detection. Rather than using external methods to detect whether a model’s output is factually correct, UQ asks how confident the model is in its own predictions. This is a form of self-assessment that reflects model reliability independent of ground truth. 35 formalize this distinction, redefining hallucination as a confident error and arguing that models should express honest uncertainty rather than optimize solely for accuracy. Consequently, token-level UQ offers a more actionable pathway for model alignment and refinement. The prevailing assumption is that UQ requires generation: sample the model’s outputs multiple times and treat divergence as a proxy for uncertainty. This method works for short question-answering tasks, but is unsuitable for clinical settings, where generating new clinical notes is hard to evaluate reliably, repeated sampling over long documents is computationally expensive, and expert-annotated clinical text is scarce. Traditionally, token-level probing is designed for masked language models like BERT ( 5 ) , which evaluate confidence by predicting a masked token within a bidirectional context. Decoder-only LLMs operate autoregressively and do not natively support such token-masking evaluation. We bridge this architectural gap with a prompt-based masking strategy. We sequentially replace each target token with a mask placeholder and prompt the model to predict it from the context. The name Reverse Probing reflects a deliberate inversion of the standard UQ paradigm. Conventional methods ask the model to generate text and then evaluate the output. We reverse this direction: instead of examining what the model produces, we feed existing clinical text back into the LLM to extract a comprehensive map of its internal activations. A well-supported token inherently aligns with and anchors to the relevant clinical records within the representation space. Conversely, when a token is unsupported, this factual grounding is absent. We harness this absence of alignment as a latent uncertainty signal, effectively turning the text itself into a probe of the model rather than a product of it. Our contributions are as follows: • We introduce Reverse Probing, the first token-level UQ framework for clinical summaries. Our framework eliminates multi-sampling to reach an exceptional computational efficiency. • We establish the first benchmark for UQ in clinical summaries. We conduct a systematic evaluation comparing Reverse Probing against eight baselines across six LLMs on two expert-annotated datasets. • We provide interpretable analysis of uncertainty sources, finding that delta energy and neighborhood context are the most universal predictors across all models and configurations. Figure 1: Overview of the Reverse Probing framework. The BHC and clinical summary are fed into a frozen LLM, which extracts four categories of features from its internal representations. A supervised classifier is then trained to predict token-level uncertainty. 2 Related Work 2.1 Uncertainty Quantification Traditionally, uncertainty quantification relies on sampling-based approaches. These methods require the model to generate multiple output candidates; the uncertainty is quantified from the variance or distribution of the generated text. SelfCheckGPT ( 25 ) , Semantic Uncertainty ( 18 ) , and Semantic Entropy ( 8 ) follow this paradigm. 12 further decomposes uncertainty into aleatoric and epistemic components. However, these methods require task-level new text generation without localizing uncertainty to specific tokens or spans. 21 show that hidden activations encode uncertainty information, and 32 show that some attention heads are correlated to incorrect generations. 9 further motivates this direction by showing that output probabilities alone are unreliable due to systematic overconfidence. Despite these advances, all existing methods are designed and evaluated on general-domain tasks. No method addresses uncertainty in long-form medical summaries, where errors are sparse, terminology is specialized, and the cost of multi-sampling is high. 2.2 Clinical Explainability and Faithfulness Explainability methods have been widely applied to clinical decision support, but rarely to uncertainty itself. SHAP ( 23 ) remains the standard framework for feature-level explanation, while 13 shows that natural language explanations outperform raw SHAP scores for clinical decision-making. 29 find that no single XAI method dominates across all clinical criteria. Several works argue that combining UQ with explainability improves reliability in high-stakes settings ( 30 ; 7 ; 22 ) . However, none of the work applies this combination specifically in clinical text. Faithfulness evaluation for clinical summaries also presents a related challenge. Standard metrics correlate poorly with expert judgement ( 1 ) , and DocLens ( 34 ) improves this with claim-level evaluation. 4 validate LLM-as-a-Judge as a scalable alternative to human annotation. 33 uses semantic consistency across sampled outputs as an uncertainty signal for radiology reports, but their approach still depends on sampling and operates at the sentence level. Token-level faithfulness evaluation over clinical text remains an open problem. 3 Methodology In clinical summaries, the vast majority of tokens are factually supported, while unsupported content appears only in sparse, scattered spans. This makes fine-grained UQ fundamentally different from task-level evaluation, where standard metrics are too coarse to capture localized errors. Our approach is inspired by masked language model probing. BERT-style models support token masking, but decoder-only LLMs are forwarding models by design. We bridge this gap with a prompt-based masking strategy: we replace each target token with a mask placeholder and prompt the model to predict it from context, turning a generative model into a token-level probe without modifying its weights. The key insight behind Reverse Probing is that the BHC serves as patient-specific evidence. When a summary token is well-supported, the model’s attention anchors to the BHC. When a token is unsupported, this shift is absent or misdirected. By comparing activations with and without the BHC, we use the absence of grounding as one of the most important signals. This is precisely where the "reverse" in Reverse Probing comes from. While traditional methods generate new text to check