Delta - Contrastive Decoding Mitigates Text Hallucinations in Large Language Models
Cheng Peng Huang, Hao Yuan Chen (Mark Chen) · 2025
Read Delta - Contrastive Decoding Mitigates Text Hallucinations in Large Language Models by Cheng Peng Huang, Hao Yuan Chen (Mark Chen) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05825.txt.
Introduction
Cheng Peng Huang and Hao-Yuan Chen curb chatbot fabrications by making the model argue with a masked twin of itself, in a 2025 paper. Their method, Delta, needs no retraining and no extra data. At answer time it hides random parts of the question and runs the model on both the full and the veiled text. Where the two runs agree, the text is likely grounded; where they split, invention creeps in.
The contrast then steers the final wording away from the shaky parts. On fact-heavy question tests the gains were clear, a few points on some sets and larger jumps on others. The sharpest win came on questions that should get no answer: the model learned to say so far more often. The authors pitch Delta as cheap and scalable, since it spends only extra inference passes. High-stakes fields like health and law stand to gain most.
Catalog record
| CREATOR | Cheng Peng Huang, Hao Yuan Chen (Mark Chen) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05825
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Delta - Contrastive Decoding Mitigates Text Hallucinations in Large Language Models Cheng Peng Huang, Hao Yuan Chen (Mark Chen) Abstract Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing tasks. Still, they are prone to generating hallucinations—factually incorrect or fabricated content that can undermine their reliability, especially in high-stakes domains such as healthcare and legal advisory. In response to this challenge, we propose Delta, a novel inference-time approach that leverages contrastive decoding to mitigate hallucinations without requiring model retraining or additional training data. Delta works by randomly masking portions of the input prompt, then contrasting the original and masked output distribution generated by the model, effectively mitigating hallucinations through inference-only computations. Delta was evaluated on context-rich QA benchmarks like SQuAD v1.1 and v2, achieving around 3 and 6 percentage points of improvement, respectively. It also showed gains of 7 and 2 percentage points on TriviaQA and Natural Question under-sampling decoding. Delta improved SQuAD v2’s no-answer exact match by over ten percentage points. These findings suggest that Delta is particularly effective when hallucinations arise from contextual ambiguity. Delta presents a computationally efficient and scalable solution for reducing hallucinations in real-world LLM applications by focusing on inference-time enhancements. 1 Introduction The rapid development of large language models (LLMs) Brown et al. 2020 has led to significant advancements in text generation, natural language processing, and a wide range of real-world applications OpenAI et al. 2024 . These models, powered by vast datasets and complex architectures, have become essential tools for translation, summarization, and conversational AI tasks McKenna et al. 2023 . Despite these achievements, LLMs face a critical challenge: their probabilistic and nondeterministic nature often generates ”hallucinated” content Xu et al. 2024 . These hallucinations manifest as text that may sound plausible but is factually incorrect or fabricated. This poses a significant issue, particularly in high-stakes domains such as healthcare, legal advisory, and scientific research, where the accuracy and reliability of the generated content are paramount. Hallucinations in LLMs arise from their reliance on patterns learned during training, causing them to occasionally generate outputs unsupported by the input data or real-world facts Huang et al. 2023 . Addressing this issue is crucial for improving the reliability and trustworthiness of LLMs, especially as they are increasingly integrated into real-time systems and applications where incorrect information can lead to severe consequences. In response to this challenge, we introduce Delta, a novel approach designed to identify and mitigate text hallucinations on inference-time computation. Unlike traditional methods Ji et al. 2023 ; Li et al. 2023b ; Ouyang et al. 2022 focusing on retraining models or requiring access to additional data, Delta operates solely at inference time, making it computationally efficient and easily deployable in real-time systems. Delta’s core innovation lies in its use of contrastive decoding Li et al. 2023a ; Chuang et al. 2024 , which leverages masked versions of the input text to contrast plausible outputs against potentially hallucinated ones. Delta builds upon previous work in vision-language models, particularly the method introduced in Leng et al. 2024 , which mitigates object hallucinations by applying Gaussian noise to the visual input during contrastive decoding. However, adapting this approach to text is more challenging, as directly applying noise to textual input is not feasible. To address this, Delta employs a masking strategy Wettig et al. 2023 , where tokens in the input sequence are randomly masked to simulate ambiguity. Delta can filter out hallucinated content more effectively during inference by comparing the model’s predictions on masked versus unmasked inputs. Our experimental results demonstrate the effectiveness of the Delta method, achieving notable improvements in question-answering accuracy. Specifically, Delta delivered gains of approximately 3 and 6 percentage points on SQuAD v1.1 and v2 Rajpurkar et al. 2016 , respectively. It achieved a 14.53 percentage point improvement in the no-answer exact match score on SQuAD v2. For more challenging QA datasets like TriviaQA Joshi et al. 2017 and Natural Questions Kwiatkowski et al. 2019 , Delta achieved enhancements of 7 and 2 percentage points under-sampling decoding compared to the baseline. These results confirm Delta’s robustness and effectiveness in context-rich datasets, showcasing its ability to mitigate hallucinations and enhance performance. However, a fundamental limitation of Delta is its marginal effectiveness on tasks without explicit contextual information. For instance, datasets like CommonsenseQA Talmor et al. 2019 and MMLU, which rely heavily on general or implicit knowledge, showed minimal or no improvements, indicating the method’s specific suitability for context-driven scenarios. 2 Related Works Recent studies have focused on mitigating hallucinations in large language models (LLMs) and large vision-language models (LVLMs) Hinck et al. 2024 , where models tend to generate inaccurate or irrelevant outputs. In vision-language models, such hallucinations often result from over-reliance on language priors or biases embedded in the datasets. For example, in LVLMs, object hallucinations occur when models predict objects not present in the image due to biased object co-occurrences in the training data. Several approaches have been developed to address this, including Visual Contrastive Decoding (VCD) and Instruction Contrastive Decoding (ICD). The Visual Contrastive Decoding (VCD) method aims to reduce object hallucinations by contrasting the outputs generated from original and distorted visual inputs. This approach does not require additional training or external pre-trained models, making it a computationally efficient solution. By introducing visual uncertainty, such as Gaussian noise, the method identifies and mitigates instances where the model overly relies on language priors or statistical biases from the training data, thereby reducing hallucinations Leng et al. 2024 . Similarly, the Instruction Contrastive Decoding (ICD) technique is employed to tackle hallucinations in multimodal tasks by incorporating instruction disturbances. This method manipulates the confidence of multimodal alignment in the model’s visual and textual inputs, helping it differentiate between hallucinated and relevant tokens. By applying contrastive penalties to tokens influenced by instruction disturbances, ICD effectively reduces the generation of hallucinated outputs, especially in complex visual contexts Leng et al. 2024 . In addition, the work context-aware decoding (CAD) has Shi et al. 2024 demonstrated a similar outcome to our Delta method by adjusting the output probabilities of LMs, amplifying the differences between outputs generated with and without the given context. This contrastive approach encourages models to prioritize contextual information during text generation. Notably, CAD can be applied to pre-trained LMs without additional training. Unlike our Delta method, the method is mainly based on context-driven datasets, making it less generalizable than the Delta method, which, in theory, could apply to all textual inputs. Both approaches are part of a growing body of work exploring contrastive mechanisms and fine-grained multimodal alignment techniques to mitigate hallucinations in models that integrate vision and language processing. Future research will likely explore more robust mechanisms further to improve model reliability across different types of multimodal tasks. 3 Method The work introduces Delta, a novel method that effectively mitigates hallucinations in text-based large-language models. The core idea behind Delta is to address the issue of hallucinations by manipulating the inference process itself. Specifically, the method generates output tokens from the model using a standard inference procedure, as outlined in Equation 1 . Building on hypotheses inspired by Leng et al. 2024 , which suggest that incomplete prompting or missing information tends to amplify hallucination effects, Delta