Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

Shiyu Ni Keping Bi Jiafeng Guo Xueqi Cheng · 2025

Read Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity by Shiyu Ni Keping Bi Jiafeng Guo Xueqi Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17537.txt.

Introduction

Shiyu Ni, Keping Bi, Jiafeng Guo, and Xueqi Cheng investigate why large language models answer wrongly with high confidence in this two thousand twenty-five paper. Their lens is knowledge popularity, measured on factual questions built from Wikidata triplets. They track the popularity of the question entity, the popularity of the answer, and how often the two co-occur. Two consistent patterns emerged from the analysis.

First, hallucinated answers are not random: they tend to be more popular, or more strongly associated with the question, than the correct answers. Second, confidence tracks answer popularity even among wrong predictions. Adding popularity signals to confidence estimation sharply cut average confidence on wrong answers and improved calibration across six models and three datasets.

Catalog record

CREATORShiyu Ni Keping Bi Jiafeng Guo Xueqi Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17537

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity Shiyu Ni Keping Bi Jiafeng Guo Xueqi Cheng Abstract Large language models (LLMs) often produce incorrect answers with high confidence, yet the factors associated with such overconfidence remain insufficiently understood. We study this problem through the lens of knowledge popularity. Using entity-centric factual QA derived from Wikidata triplets, we characterize popularity through question entity popularity, answer popularity, and question–answer co-occurrence. We find two consistent patterns. First, hallucinated answers are far from random: compared with ground-truth answers, they tend to be more popular or more frequently associated with the question entity. Second, confidence is strongly tied to the popularity of generated answers: even among incorrect predictions, more popular answers or those with higher question–answer co-occurrence receive higher confidence. Together, these findings suggest that popular but wrong alternatives may contribute to overconfidence. We further show that popularity-related signals can mitigate overconfidence and improve overall confidence estimation. Across six models and three datasets, incorporating knowledge popularity reduces average confidence on incorrect answers from 0.765 to 0.254 and overall ECE from 0.356 to 0.050, while improving Alignment from 77.08% to 83.72%. 1 Introduction A trustworthy model should be well calibrated: its confidence in an answer, measured by the generation probability assigned to that answer, should reflect the likelihood that the answer is correct. However, large language models (LLMs) have been shown to be overconfident, often assigning high confidence to incorrect answers ( Jiang et al., 2021 ; Xiong et al., 2023 ; Ni et al., 2024a ; Yang et al., 2023 ) . Such miscalibrated confidence poses significant risks, especially in safety-critical domains such as healthcare and finance. Despite extensive evidence of LLM overconfidence, why incorrect answers can receive high confidence remains insufficiently understood. Figure 1: An example where the model generates an incorrect answer that is more popular than the correct answer, while assigning it high confidence. Overconfidence can be viewed as a mismatch between the correctness of a generated answer and the confidence assigned to it. We suspect that one source of this mismatch lies in the frequency structure of the knowledge learned by LLMs. Repeated exposure to an entity may make it more likely to be generated, but such popularity does not guarantee that the entity is correct for the queried relation. This motivates us to study overconfidence through the lens of knowledge popularity. Knowledge popularity has been widely studied as an important factor in LLM factual recall. Prior work shows that LLMs are more likely to answer correctly when the required knowledge appears more frequently in the corpus ( Mallen et al., 2023 ; Kandpal et al., 2023 ) . However, these studies mainly focus on the ground-truth side, explaining when models are likely to know the correct answer. Overconfidence, in contrast, concerns the confidence assigned to the generated answer, especially when that answer is incorrect. We therefore shift attention from ground-truth-side popularity to generated-answer-side popularity signals. Specifically, we ask two questions: (1) When LLMs make mistakes, what kinds of answers do they generate? (2) How are generated-answer-side popularity signals associated with model confidence, especially among incorrect answers? To answer these questions, we focus on factoid question answering (QA), which allows us to characterize generation-side popularity through question entity popularity, generated answer popularity, and question-answer co-occurrence. We employ three representative datasets constructed by prior work ( Yuksekgonul et al., 2023 ) , with examples shown in Figure 3 . As the models’ training data are not accessible, we follow prior studies ( Mallen et al., 2023 ; Kandpal et al., 2023 ; Yuksekgonul et al., 2023 ) and use external statistics as proxies, including Wikidata sitelinks for entity popularity and Wikipedia document co-occurrence for question-answer co-occurrence frequency. Our results reveal a systematic popularity-related pattern behind overconfident errors. When factual recall fails, LLMs do not generate arbitrary wrong answers; instead, their errors tend to gravitate toward entities that are more popular or co-occur more frequently with the question entity than the ground-truth answer. More importantly, popularity is also related to how strongly models trust these errors: even among incorrect predictions, answers with stronger popularity signals tend to receive higher confidence. Together, these findings suggest that popular but wrong alternatives may contribute to overconfidence. The pattern is particularly evident when models struggle with the task and persists across model scales. Although popular incorrect answers can receive high confidence, correct predictions typically exhibit even stronger and more reliable question-answer association patterns. We therefore leverage these signals for confidence calibration. Across six models and three datasets, incorporating knowledge popularity substantially mitigates overconfidence, reducing the average confidence on incorrect answers from 0.765 to 0.254, while improving overall confidence calibration, with ECE decreasing from 0.356 to 0.050 and Alignment increasing from 77.08% to 83.72%. To reduce reliance on external corpora, we further explore using the model itself to estimate popularity signals, which remains a reasonable alternative. Figure 2: Overview of knowledge popularity, overconfidence, and calibration. The upper panel illustrates overconfident errors involving popular or frequently co-occurring incorrect answers; the lower panels show correct high-confidence predictions and popularity-aware confidence calibration. 2 Related Work Existing research on model knowledge boundary perception focuses on assessing model confidence and can be mainly classified into four categories. Probabilistic Confidence. This line of research uses the probability assigned to an answer as a measure of model confidence ( Guo et al., 2017 ; Desai and Durrett, 2020 ; Jiang et al., 2021 ; Kadavath et al., 2022 ; Si et al., 2022 ; Kuhn et al., 2023 ) . Guo et al. (2017) examined modern neural networks (e.g., ResNet ( He et al., 2016 ) ) and found them to be overconfident, proposing temperature scaling as a remedy. Later, Desai and Durrett (2020) showed that BERT-style models tend to be relatively well-calibrated, while Jiang et al. (2021) found that pre-trained language models such as T5 ( Raffel et al., 2020 ) remained overconfident. More recent work has turned to LLMs, with studies showing that they, too, exhibit overconfidence ( Si et al., 2022 ; Lin et al., 2022 ; Tian et al., 2023 ) . Verbalized Confidence. LLMs have been shown to express their confidence verbally ( Lin et al., 2022 ; Yin et al., 2023 ; Tian et al., 2023 ; Xiong et al., 2023 ; Yang et al., 2023 ; Ni et al., 2024a ) . Some studies ( Yin et al., 2023 ; Ni et al., 2024a ) found that LLMs often fail to recognize their knowledge limitations verbally and tend to be overconfident. Xiong et al. (2023) systematically studied black-box approaches for estimating LLM confidence. Beyond prompting-based methods, some studies aim to train LLMs to verbalize more accurate confidence ( Lin et al., 2022 ; Yang et al., 2023 ; Zhang et al., 2024 ) . Consistency-based Confidence. If the model is confident in its answer, it should maintain consistency across multiple generations. Recent studies have used self-consistency across generations as a proxy for LLM confidence ( Manakul et al., 2023 ; Kuhn et al., 2023 ) . Zhang et al. (2023) extended this by evaluating the consistency of answers across multiple semantically equivalent inputs and across different models. Ding et al. (2024) further adapted this approach

Read the complete text