When an LLM is apprehensive about its answers - and when its uncertainty is justified

Petr Sychev, Andrey Goncharov, Daniil Vyazhev, Edvard Khalafyan, Alexey Zaytsev · 2025

Read When an LLM is apprehensive about its answers - and when its uncertainty is justified by Petr Sychev, Andrey Goncharov, Daniil Vyazhev, Edvard Khalafyan, Alexey Zaytsev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01688.txt.

Introduction

Petr Sychev and four co-authors present a 2025 study of when a language model's own doubt signals that its answer is wrong. Knowing when to trust a model matters most where errors are costly. The team tested two doubt measures: the spread of the model's token choices, and a setup where a model judges another's answer. They ran three model families, in sizes from 1.5 to 72 billion parameters, across 14 topics of multiple-choice questions.

The judging approach predicted errors no better than chance. Token spread worked well on knowledge-heavy topics such as biology, where it also flagged hard questions. On math questions, which hinge on reasoning steps, the signal nearly vanished. The authors argue that doubt measures must account for how much reasoning a question needs, and note bias in the test set itself.

Catalog record

CREATORPetr Sychev, Andrey Goncharov, Daniil Vyazhev, Edvard Khalafyan, Alexey Zaytsev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When an LLM is apprehensive about its answers - and when its uncertainty is justified Petr Sychev, Andrey Goncharov, Daniil Vyazhev, Edvard Khalafyan, Alexey Zaytsev Abstract . Uncertainty estimation is crucial for evaluating Large Language Models (LLMs), particularly in high-stakes domains where incorrect answers result in significant consequences. Numerous approaches consider this problem, while focusing on a specific type of uncertainty, ignoring others. We investigate what estimates, specifically token-wise entropy and model-as-judge (MASJ), would work for multiple-choice question-answering tasks for different question topics. Our experiments consider three LLMs: Phi-4, Mistral, and Qwen of different sizes from 1.5B to 72B and 14 14 topics. While MASJ performs similarly to a random error predictor, the response entropy predicts model error in knowledge-dependent domains and serves as an effective indicator of question difficulty: for biology ROC AUC is 0.73 0.73 . This correlation vanishes for the reasoning-dependent domain: for math questions ROC-AUC is 0.55 0.55 . More principally, we found out that the entropy measure required a reasoning amount. Thus, data-uncertainty related entropy should be integrated within uncertainty estimates frameworks, while MASJ requires refinement. Moreover, existing MMLU-Pro samples are biased, and should balance required amount of reasoning for different subdomains to provide a more fair assessment of LLMs performance. \english 1. Introduction LLMs have succeeded in various tasks, demonstrating impressive capabilities in generating human-like text and solving complex problems. However, their ability to accurately assess uncertainty in their predictions remains a significant challenge [ 17 ] . This gap between a model’s confidence and its actual correctness, often called the confidence gap [ 5 ] , poses critical risks, particularly in high-stakes applications where overconfidence in incorrect answers leads to severe consequences: in domains such as health, law [ 7 ] , education, or economics an LLM’s overestimation of its accuracy could result in harmful decisions or misinformation. Understanding and decreasing this confidence gap is therefore essential for ensuring the safe and reliable deployment of LLMs. Addressing this limitation has led to the development of numerous uncertainty estimation techniques. Most of them belong to one of two problem statements. They either focus on binary classification or rely on rich signals from long, free-text responses [ 26 ] . These methods often generalize poorly for concise multiple-choice questions. Figure 1 . Question complexity evaluation pipeline. See more details in the methods section Moreover, existing methods tend to answer how good an overall confidence estimate is, ignoring why uncertainty appeared in a specific case. While classic decomposition of uncertainty to aleatoric (data) and epistemic (model) [ 15 ] also holds for LLMs, the problem is challenging [ 14 ] . For example, the authors in [ 1 ] consider only epistemic uncertainty. To address these gaps, we propose a pipeline that enables detailed investigation of domain-specific or complexity-labeled datasets with multiple-choice answers. This framework allows us to explore how commonly used uncertainty estimation methods, particularly entropy-based approaches, perform under varying question topics and levels of required reasoning. We can also leverage an auxiliary LLM to generate labels for reasoning, knowledge requirements, and the number of reasoning steps. The introduction of the pipeline led to a more detailed investigation of datasets commonly used for benchmarking LLMs and their uncertainty estimates. In more detail, our main contributions are the following: • We present a fully automated pipeline for evaluating uncertainty estimation approaches within the MMLU-Pro setting. It is presented in Figure 1 and allows datasets or tested methods to be varied. The associated work is available on Github 1 1 1 https://github.com/LabARSS/question-complextiy-estimation . • We propose using data uncertainty estimates based on token-wise entropy and model uncertainty estimates based on MASJ. Thus, our approach covers two main components of uncertainty estimation: data and model one, given that MMLU-Pro answers are typically short and provide limited information on model confidence. • Within the developed pipeline, we consider different subsets of the MMLU-Pro dataset with splits by domain and reasoning requirement estimated via MASJ proposed in this work. • Our experimental evidence that includes consideration of four models (Phi-4, Mistral-Small-24B-Instruct, Qwen-1.5 B, Qwen-72B) suggest that the entropy predicts well errors of an LLM if the amount of required reasoning is small. Moreover, its quality increases if the model size increases. Unlike MASJ provides much weaker results, being unable to identify error patterns. Further progress here would be possible with more reasoning steps during an uncertainty estimate with an entropy-based approach on top of it. • We also observe that the existing dataset MMLU-Pro has internal biases with the complexity of the question, the same as in [ 12 ] , and the required reasoning amount for them varies significantly depending on the topic. Thus, more advanced and less biased datasets are required for a fair measurement of the progress of LLMs. Additional human annotation of questions in a dataset could identify further gaps in LLMs’ capabilities. 2. Related Works Tests have been a popular way to estimate an individual’s proficiency for more than 500 years [ 3 ] . After their emergence, LLMs have also been tested extensively and compared to each other or humans [ 22 ] . Several papers established benchmarks, with natural ones being QA datasets: an LLM receives a prompt and should generate a response citerogers2023qa. Different questions exist, but automated evaluation of answers is available only for multiple-choice questions. For LLMs, slight changes in the way the questions are asked can significantly affect the accuracy of a model, probably due to data leakage and format overfitting [ 8 ] , other options are much more expensive and challenging. One of the recent options that takes into account possible leakages - and, on the other hand, is widely used is an expanded version of the MMLU dataset, called MMLU-Pro [ 24 ] with about 0.56 accuracy for Llama-70B model. The presented \sim 12000 questions span 13 diverse topics, including STEM and other areas of scientific studies. The complexity, volume, and diversity of the dataset provide a strong foundation for our study of how different aspects of the questions affect the probability of correctly answering them. In related works, datasets such as ARC-AI2 [ 6 ] and GPQA [ 20 ] have been widely used for evaluating multiple-choice QA in the scientific domain as well. However, ARC-AI2 is often criticized for its relatively low difficulty, while GPQA, though highly challenging, suffers from a dataset size of fewer than 500 samples, which restricts its utility for robust evaluation. To address these limitations and ensure a comprehensive assessment, we focus our final experiments on MMLU-Pro, which offers a larger scale and a more balanced difficulty level for evaluating model performance in scientific reasoning tasks. MMLU-Pro is suited for evaluating a model’s ability to identify uncertainty in the generated output and signal potential errors, which addresses a fundamental question in AI safety. In a broader context, we can refer to this effect as hallucination [ 14 ] ; in a more classic context, it is a prediction error [ 10 ] . Another related approach is assessing the question difficulty, provided in [ 11 ] , but suitable there only for an RAG scenario. Universal Model-as-judge (MASJ) approaches use available models or a bigger and more powerful model to

Read the complete text