Bolster Hallucination Detection via Prompt-Guided Data Augmentation

Wenyun Li, Zheng Zhang, Dongmei Jiang, Xiangyuan Lan · 2025

Read Bolster Hallucination Detection via Prompt-Guided Data Augmentation by Wenyun Li, Zheng Zhang, Dongmei Jiang, Xiangyuan Lan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-15977.txt.

Introduction

Wenyun Li, Zheng Zhang, Dongmei Jiang, and Xiangyuan Lan make training data to catch fabricated model output. Detectors for hallucinations need labelled cases of both true and invented text. Such data are scarce and costly to build by hand. Their framework, PALE, uses prompt-guided replies from language models as cheap extra data for the detector.

Prompts steer the generator to produce true and hallucinated samples on demand. No extra human labelling is needed at any stage. To judge the sparse internal vectors such text leaves, they add a new score. The Contrastive Mahalanobis Score models how the two classes spread in activation space. A matrix split helps it capture that structure well. Across wide tests, PALE beats the rival baseline by a clear margin in detection scores. The four authors stress its broad reach and fit for real use.

Catalog record

CREATORWenyun Li, Zheng Zhang, Dongmei Jiang, Xiangyuan Lan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-15977

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bolster Hallucination Detection via Prompt-Guided Data Augmentation Wenyun Li, Zheng Zhang, Dongmei Jiang, Xiangyuan Lan Abstract Large language models (LLMs) have garnered significant interest in AI community. Despite their impressive generation capabilities, they have been found to produce misleading or fabricated information, a phenomenon known as hallucinations. Consequently, hallucination detection has become critical to ensure the reliability of LLM-generated content. One primary challenge in hallucination detection is the scarcity of well-labeled datasets containing both truthful and hallucinated outputs. To address this issue, we introduce P rompt-guided data A ugmented ha L lucination d E tection (PALE), a novel framework that leverages prompt-guided responses from LLMs as data augmentation for hallucination detection. This strategy can generate both truthful and hallucinated data under prompt guidance at a relatively low cost. To more effectively evaluate the truthfulness of the sparse intermediate embeddings produced by LLMs, we introduce an estimation metric called the Contrastive Mahalanobis Score (CM Score). This score is based on modeling the distributions of truthful and hallucinated data in the activation space. CM Score employs a matrix decomposition approach to more accurately capture the underlying structure of these distributions. Importantly, our framework does not require additional human annotations, offering strong generalizability and practicality for real-world applications. Extensive experiments demonstrate that PALE achieves superior hallucination detection performance, outperforming the competitive baseline by a significant margin of 6.55%. Code — https://github.com/li-wenyun/PALE Introduction Recently, large language models (LLMs) have emerged as one of the most significant breakthroughs in the field of artificial intelligence ( Hurst et al. 2024 ) . LLMs have found widespread application across a variety of tasks, including logical reasoning ( Dong et al. 2024 ) , visual question answering ( Jian, Yu, and Zhang 2024 ) , and speech-to-text transcription ( Zhang et al. 2023 ) , often surpassing human performance in many scenarios. Due to their exceptional reasoning and generative capabilities, LLMs have been integrated into numerous high-trust systems, such as those in the medical domain ( Liu et al. 2023 ) . However, despite these impressive capabilities, even state-of-the-art LLMs frequently generate factually incorrect or nonsensical content—a phenomenon known as hallucination ( Huang et al. 2025 ) . This inherent risk makes LLM outputs potentially unreliable in mission-critical applications ( Wang et al. 2024b ) . Therefore, a reliable LLM should not only produce text that is coherent with the given prompt but also possess the ability to detect hallucinations in its output. This concern underscores the importance of hallucination detection technology, which determines whether a generated output is truthful or not ( Chen et al. 2023 ; Sriramanan et al. 2024 ; Su et al. 2024 ) . Most approaches to hallucination detection rely on devising uncertainty scoring functions. For instance, logit-based methods ( Ren et al. 2023 ; Malinin and Gales 2021 ) employ token-level probabilities as uncertainty scores, consistency-based methods ( Lin, Trivedi, and Sun 2024a ; Manakul, Liusie, and Gales 2023 ) quantify uncertainty by evaluating the consistency across multiple responses, and verbalized methods ( Lin, Trivedi, and Sun 2024a ; Kadavath et al. 2022 ) prompt LLMs to express their uncertainty in natural language. Recently, there has been increasing work ( Burns et al. 2022 ; Du, Xiao, and Li 2024 ; Arteaga, Schön, and Pielawski 2024 ) aimed at leveraging the internal states of LLMs to assess the veracity of their outputs. For example, contrast-consistent search (CCS) ( Burns et al. 2022 ) trains a binary truthfulness classifier to satisfy logical consistency properties, and HaloScope ( Du, Xiao, and Li 2024 ) seeks to identify hallucination subspaces. However, all these methods face the challenge of a scarcity of well-labeled datasets containing both truthful and hallucinated outputs, which significantly limits their performance in detecting hallucinations in real-world scenarios. We argue that the primary challenge in hallucination detection is the lack of labeled datasets containing both truthful and hallucinated generations. In practice, creating a high-quality ground truth dataset for hallucination detection requires substantial human labor to annotate a large number of generated samples. However, collecting such large-scale data and labeling it is extremely costly. In particular, the vast landscape of generative models and the diverse range of content they produce further complicate the process. Moreover, maintaining the quality and consistency of labeled data for hallucination detection requires continuous annotation efforts and robust quality control measures. These formidable obstacles underscore the need to reduce reliance on human-labeled data for hallucination detection. Figure 1: The pipeline of our proposed PALE. Our PALE comprises three steps: 1) Utilizing state‑of‑the‑art LLMs to generate both truthful and hallucinated data; 2) Employing the Contrastive Mahalanobis Score to determine the truthiness of test samples. To address this issue, we introduce P rompt-guided data A ugmented ha L lucination d E tection (PALE), a novel framework that leverages state-of-the-art LLMs for data augmentation using non-parametric prompts. The augmented data generated through LLM prompting can be acquired at a relatively low cost. A state-of-the-art language model, such as GPT-4o, contains a vast amount of hidden knowledge, enabling us to harness LLMs to produce diverse truthful and hallucinated data via prompt engineering. From this perspective, we introduce a novel metric, the Contrastive Mahalanobis Score , to assess the truthfulness of augmented data. Our key idea is inspired by prior work ( Du, Xiao, and Li 2024 ; Burns et al. 2022 ) , which leverages the latent state of a language model as a representation that encodes information related to truthfulness. Specifically, PALE employs a matrix decomposition approach to model the distributions of truthful and hallucinated data in the embedding activation space. It then computes the distance of a test sample to both distributions. Both truthful and hallucinated distributions are modeled as Gaussian, with their means and covariance matrices estimated from the embedding activations. Notably, the covariance matrix computation is facilitated by factorizing the LLM embeddings. The Mahalanobis distance between a test sample and each distribution is used to infer its truthfulness. The resulting score is straightforward to implement in practical applications. The primary contributions of our research are summarized as follows: • Our proposed framework, P rompt-guided data A ugmented ha L lucination d E tection (PALE), harnesses augmented data generated through prompt engineering for hallucination detection in LLMs. This approach not only yields significant performance improvements but also reduces reliance on costly human annotation. • We present a scoring function called Constrastive Mahalanobis Score , which leverages the distances to both truthful and hallucinated data distributions in the activation space to effectively determine the truthfulness of test data. • Extensive experiments demonstrate the superior effectiveness of our method compared to other state-of-the-art approaches. Moreover, comprehensive ablation studies assess the impact of various design choices in PALE and confirm its scalability across larger LLMs and diverse datasets. These findings provide a systematic and thorough understanding of leveraging LLM-augmented data for hallucination detection, paving the way for future research. Related Works Hallucination in LLMs Large Language Models (LLMs) have made remarkable achievements across various AI domains, including logical reasoning ( Dong et al. 2024 ; Wang et al. 2024a ) , visual question answering ( Jian, Yu, and Zhang 2024 ) , text generation ( Min et al. 2023 ) , and speech-to-text transcription

Read the complete text