Black-Box Hallucination Detection via Consistency Under the Uncertain Expression

2025

Read Black-Box Hallucination Detection via Consistency Under the Uncertain Expression (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21999.txt.

Introduction

Seongho Joo, Kyungmin Min, Jahyun Koo, and Kyomin Jung propose a simple way to spot made-up answers from language models in this 2025 paper. Most detectors need outside resources or a peek inside the model, such as word probabilities. Closed models expose neither. The authors look for a black-box signal, one that uses only the text the model returns.

Their signal comes from how models behave when asked to express uncertainty. The study found that factual answers stay consistent under such phrasing, while invented answers shift around. The team turns that gap into a detection metric. In experiments, the metric predicted the factuality of responses better than baselines that did use internal model information. The result is a practical check that works through ordinary interfaces, on models whose insides are hidden.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-21999

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Black-Box Hallucination Detection via Consistency Under the Uncertain Expression Abstract Despite the great advancement of Language modeling in recent days, Large Langauge Models (LLMs) such as GPT3 are notorious for generating non-factual responses, so-called "hallucination" problems. Existing methods for detecting and alleviating this hallucination problem requires external resource or the internal state of LLMs, such as the output probability of each token. Given the LLM’s restricted external API availability and the limited scope of external resources, there is an urgent demand to establish the Black-Box approach as the cornerstone for effective hallucination detection. In this work, we propose a simple black-box hallucination detection metric after the investigation of the behavior of LLMs under expression of uncertainty. Our comprehensive analysis reveals that LLMs generate consistent responses when they present factual responses while non-consistent responses in vice-versa. Based on the analysis, we propose an efficient black-box hallucination detection metric with the expression of uncertainty. The experiment demonstrates that our metric is more predictive of the factuality in model response than baselines that use internal knowledge of LLMs. Keywords: Factuality, Hallucination, LLMs Black-Box Hallucination Detection via Consistency Under the Uncertain Expression Seongho Joo, Kyungmin Min, Jahyun Koo, Kyomin Jung Seoul National University {seonghojoo, kyungmin97, koojahyun, kjung}@snu.ac.kr Abstract content 1. Introduction In recent days, Large Language Models (LLMs) have made great progress in natural language generation. Based on a vast amount of training corpus, they have shown remarkable performance in open-domain QA, abstractive summarization, and machine translation Goyal et al. 2023 ; Ouyang et al. 2022 ; Touvron et al. 2023 . To detect and mitigate the hallucination problem, many hallucination detection methods rely on external resources Shuster et al. 2021 ; Ji et al. 2022 ; Guerreiro et al. 2022 or require internal state of LLMs like probability per output token Azaria and Mitchell 2023 ; Kuhn et al. 2023 , and few black-box methods that only require the response of LLMs are proposed Manakul et al. 2023 ; Cohen et al. 2023 . However, we often do not have access to the internal information of LLMs or external sources but only their generated output. Furthermore, external knowledge sources like Wikipedia may not cover sufficient knowledge to answer the question. It is also reported that using the retriever model for question answering may worsen the performance when dealing with factual knowledge in long-tail Mallen et al. 2023 . Previous black-box hallucination detection methods rely on obtaining around ten LLM responses for successful detection Manakul et al. 2023 . These sampling-based methods come with significant computational costs and can pose challenges in determining the appropriate sampling temperature for detection. However, our approach necessitates just a single extra model inference. Figure 1 : Example of prompt-based hallucination detection without an external source. Given the question, the expression of uncertainty is used as the prefix for the answer part. The answer before the prompt perturbation and the response with the new prompt are compared. In this work, we aim to develop an efficient prompt-based hallucination detection method that does not require multiple sampling of responses. We explore prompts with expressions that can discriminate between factual responses and non-factual responses generated by LLMs as shown in Figure 1 . Motivated by the fact that humans express their level of confidence when answering questions, our preliminary study examines what happens when LLMs express their level of confidence in their answers. Therefore, we investigate the following question: "Do LLMs behave consistently or differently under expression of uncertainty when they know the answer?". To answer this question, we gather expressions of uncertainty such as "I am not sure but it could be" and expressions of certainty, such as "It must be." These expressions serve as prompts for our investigation like Figure 1 . Through the experiment, we aim to answer the following question: • Do LLMs generate consistent responses under uncertain or certain expressions? • Is there a difference in terms of response accuracy or confidence of LLMs between when the model answers with uncertain expression and certain expression? In Section 3 , we conduct a comprehensive analysis regarding the questions on two open-domain question answering (QA) benchmarks, HotpotQA Yang et al. 2018 , and NQ-open Kwiatkowski et al. 2019 in the closed-book setting. We evaluate the performance of GPT-3 text-davinci-003 in a zero-shot manner and divide the (question, answer) pair into two groups. Specifically, we divide the samples into the factual group and the non-factual group based on the factuality of the original response before the prompt perturbation. In both datasets, there is a clear distinction between the two groups when the expression of uncertainty is used. Most samples in the factual group show consistency in the response, while samples of the non-factual group often show inconsistency in the response after the prompt is changed. In addition, we also find the number of samples that show consistency differs not much in the response between when uncertain expression and certainty are used. Based on our findings, we propose a simple hallucination detection metric using only a few number of responses of LLMs in Section 4 . Specifically, we utilize the consistency between the response generated using the standard prompt and the response generated using a prompt that includes an uncertain expression. Unlike previous detection metrics, our metric does not require the internal state of LLMs nor multiple responses through multiple sampling. The experiment demonstrates that our metric efficiently distinguishes factual answers compared to the baseline that requires internal knowledge of LLMs. 2. Related Work 2.1. Large Language Models Large Language Models (LLMs) are pre-trained on massive amounts of text corpora, and this pre-training equips LLMs with powerful text generation capabilities and zero-shot abilities. Brown et al. 2020 ; Zhang et al. 2022 ; Chowdhery et al. 2022 ; Touvron et al. 2023 . Through instruction tuning and reinforcement learning, such as reinforcement learning from human feedback (RLHF), LLMs demonstrate better alignment with user intent, resulting in improved zero-shot abilities and in-context learning Ouyang et al. 2022 ; Wei et al. 2022 ; Longpre et al. 2023 ; Chung et al. 2022 . Additionally, instruction tuning enhances LLMs’ proficiency in user interactions, leading to the emergence of practical models Liu et al. 2023 ; OpenAI 2023 ; Chiang et al. 2023 . However, practical LLMs need to be improved in detecting hallucinations due to the constraints of external APIs, which hinder the devise of hallucination detection methods relying on LLMs’ internal states Azaria and Mitchell 2023 ; Kuhn et al. 2023 . Our work proposes a black-box hallucination detection method that distinguishes factual answers more effectively than previous methods without requiring access to the internal states of LLMs. 2.2. Hallucination Detection of Large Language models One way for detecting hallucinations involves using external sources (e.g., Wikipedia) to evaluate whether the generated text contains non-factual information Thorne et al. 2018 ; Augenstein et al. 2019 ; Shuster et al. 2021 ; Ji et al. 2022 ; Guerreiro et al. 2022 . Another approach is to detect hallucinations by leveraging the internal state of LLMs. The internal state of LLMs, such as output token-level probabilities, enables the derivation of a confidence score for assessing hallucination Mielke et al. 2022 ; Kadavath et al.

Read the complete text