Calibrating LLM Confidence by Probing Perturbed Representation Stability

Reza Khanmohammadi, Erfan Miahi, Mehrsa Mardikoraem, Simerjot Kaur, Ivan Brugere, Charese H. Smiley, Kundan Thind, Mohammad M. Ghassemi · 2025

Read Calibrating LLM Confidence by Probing Perturbed Representation Stability by Reza Khanmohammadi, Erfan Miahi, Mehrsa Mardikoraem, Simerjot Kaur, Ivan Brugere, Charese H. Smiley, Kundan Thind, Mohammad M. Ghassemi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21772.txt.

Introduction

A language model can sound sure and still be wrong, so Reza Khanmohammadi and co-authors built a way to read a model's confidence from the inside. Their method, CCPS, jabs the model's final hidden states with targeted perturbations and watches how the representation wobbles. Features from that response train a small classifier to predict whether the answer is right.

They tested models from 8 to 32 billion parameters across three model families, on MMLU and MMLU-Pro, in both multiple-choice and open-ended formats. CCPS beat existing confidence methods by a wide margin, the authors report. The trick is that unstable internal states betray shaky answers.

Catalog record

CREATORReza Khanmohammadi, Erfan Miahi, Mehrsa Mardikoraem, Simerjot Kaur, Ivan Brugere, Charese H. Smiley, Kundan Thind, Mohammad M. Ghassemi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE113 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21772

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Calibrating LLM Confidence by Probing Perturbed Representation Stability Reza Khanmohammadi, Erfan Miahi, Mehrsa Mardikoraem, Simerjot Kaur, Ivan Brugere, Charese H. Smiley, Kundan Thind, Mohammad M. Ghassemi Abstract Miscalibration in Large Language Models (LLMs) undermines their reliability, highlighting the need for accurate confidence estimation. We introduce CCPS (Calibrating LLM Confidence by Probing Perturbed Representation Stability), a novel method analyzing internal representational stability in LLMs. CCPS applies targeted adversarial perturbations to final hidden states, extracts features reflecting the model’s response to these perturbations, and uses a lightweight classifier to predict answer correctness. CCPS was evaluated on LLMs from 8B to 32B parameters (covering Llama, Qwen, and Mistral architectures) using MMLU and MMLU-Pro benchmarks in both multiple-choice and open-ended formats. Our results show that CCPS significantly outperforms current approaches. Across four LLMs and three MMLU variants, CCPS reduces Expected Calibration Error by approximately 55% and Brier score by 21%, while increasing accuracy by 5 percentage points, Area Under the Precision-Recall Curve by 4 percentage points, and Area Under the Receiver Operating Characteristic Curve by 6 percentage points, all relative to the strongest prior method. CCPS delivers an efficient, broadly applicable, and more accurate solution for estimating LLM confidence, thereby improving their trustworthiness. Code — https://github.com/ledengary/CCPS Data — https://huggingface.co/datasets/ledengary/CCPS 1 Introduction Despite their impressive performance, large language models (LLMs) consistently struggle with confidence calibration Guo et al. (2017) ; Geng et al. (2024) . Their confidence—the model’s internally estimated probability that a given response is correct—frequently misaligns with actual outcomes: LLMs often assign high confidence to wrong answers and low confidence to right ones. This unreliability is particularly acute in high-stakes domains like medicine, finance, and law. For example, in a medical task like symptom extraction for cancer toxicity assessment, even if an LLM often produces correct information, it might do so with inappropriately low confidence, or conversely, express high confidence for incorrect outputs. If such confidence scores are not dependable guides to actual correctness, human experts may be forced to meticulously review every LLM-generated instance, significantly diminishing the practical benefits of automation. Accurate confidence estimation for each specific response is therefore essential, as it provides a vital mechanism for managing risk, enabling users to prioritize human oversight, selectively rely on LLM outputs, and foster more responsible and effective LLM integration. Existing approaches to LLM confidence estimation include direct self-evaluation Kadavath et al. (2022) , post-hoc adjustments Jiang et al. (2021) , internal state probing with lightweight classifiers Azaria and Mitchell (2023) ; Liu et al. (2024) , and model fine-tuning Kapoor et al. (2024b) . These methods often struggle to consistently deliver on multiple desirable properties simultaneously, namely achieving strong calibration (e.g., low Expected Calibration Error (ECE)) and high discriminative power (e.g., high Area Under the Precision-Recall Curve (AUCPR) or Area Under the Receiver Operating Characteristic Curve (AUROC)) while maintaining computational efficiency and generalizability across the diverse set of LLM architectures and families. Many methods excel in some of these desirable properties but make trade-offs in others; for instance, fine-tuning approaches like Calibration-Tuning (CT) Kapoor et al. (2024b) often achieve strong calibration in ECE but may not consistently lead in discriminative metrics like AUROC, while lightweight methods such as LitCab Liu et al. (2024) can demonstrate strong AUROC but sometimes show variable ECE performance across different LLM families. This leaves a need for more holistically effective solutions. In this work, we introduce CCPS ( C alibrating LLM C onfidence by Probing P erturbed Representation S tability), a novel method that addresses these challenges by assessing LLM confidence through the stability of its internal representations. CCPS operates on frozen base LLMs, applying targeted adversarial perturbations to the final hidden states that generate an answer’s tokens. From the LLM’s response to these perturbations, we extract a rich feature set and train a lightweight classifier to predict answer correctness. This model-agnostic probing offers an efficient confidence proxy without modifying the base LLM. Comprehensive evaluations demonstrate CCPS’s significant advantages over existing confidence estimation approaches. Tested across four modern LLMs (8B to 32B parameters, spanning three architectural families) on MMLU and MMLU-Pro benchmarks in both multiple-choice and open-ended formats, CCPS consistently achieves superior performance across key calibration (e.g., ECE, Brier score) and discrimination metrics (e.g., ACC, AUCPR, AUROC). Our findings reveal that by quantifying LLM representational stability through targeted internal perturbations, CCPS achieves substantial improvements over other state-of-the-art confidence estimation methods; for instance, CCPS reduces average ECE by approximately 55% (up to 88%) and Brier score by 21% (up to 45%), while also increasing average Accuracy (ACC) by 5 percentage points (pp) (up to +14 pp), AUCPR by 4 pp (up to +13 pp), and AUROC by 6 pp (up to +17 pp), relative to the best performing baseline. The key contributions of this work include: • A novel, model-agnostic, parameter-efficient, and scalable framework (CCPS) offering a fresh perspective on LLM confidence estimation by quantifying it through the stability of internal representations under targeted perturbations. • Demonstration of CCPS’s substantial improvements in both key calibration (ECE, Brier score) and discrimination (ACC, AUCPR, AUROC) metrics. • Evidence of CCPS’s generalizability across diverse LLM architectures (Llama, Qwen, Mistral; 8B to 32B). • Extensive benchmarking of confidence estimation methods on MMLU and MMLU-Pro in both multiple-choice and open-ended formats. These contributions establish CCPS as an effective method for improving LLM confidence estimation, helping to make LLM applications more trustworthy, especially in critical domains where reliability is crucial. 2 Related Work Calibration in LLMs A model is considered well-calibrated when its expressed confidence in a prediction aligns with the empirical likelihood of that prediction being correct. In the context of LLMs, calibration efforts broadly diverge into two streams. The first targets calibration of next-token predictions and responses to reduce hallucinations. This direction is exemplified by the work of Zhou et al. (2025) , which focuses on hallucination mitigation through comprehensive model calibration. The second stream, more aligned with the present work, focuses on developing and calibrating explicit confidence estimation mechanisms that assess the correctness of statements generated by LLMs. Confidence Estimation in LLMs Several approaches have been proposed for estimating an LLM’s confidence in its assertions. One vein of research explores probing the internal states of LLMs. For instance, Azaria and Mitchell (2023) train an auxiliary linear classifier on hidden layer activations from an LLM to predict the truthfulness of statements. While this can reveal internal knowledge, its efficacy depends on identifying the optimal representational layer and may vary across evaluation metrics. Another approach involves eliciting the model’s inherent self-assessment. Kadavath et al. (2022) introduced concepts like P(True), the probability an LLM assigns to its generated answer being correct (often derived from probabilities of “True” or “False” tokens when prompted to evaluate its own previous answer), and P(IK), the probability the model assigns to its own ability to answer a given question correctly, estimated before attempting to generate the answer. These methods assess the model’s intrinsic confidence without external classifiers but rely on the LLM’s inherent, and often uncalibrated, self-evaluation capabilities. Improving Confidence Calibration in LLMs Other research adapts the LLM or its outputs to produce more reliable confidence scores. Logit Temperature Scaling (LTS) Jiang et al. (2021) is a post-hoc method that adjusts output logits using a learned temperature parameter; however,

Read the complete text