The Mirage of Performance Gains: Why Contrastive Decoding Fails to Address Multimodal Hallucination

Hao Yin, Gunagzong Si, Zilei Wang · 2025

Read The Mirage of Performance Gains: Why Contrastive Decoding Fails to Address Multimodal Hallucination by Hao Yin, Gunagzong Si, Zilei Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10020.txt.

Introduction

Hao Yin and co-authors wrote a 2025 paper asking why contrastive decoding fails to stop object hallucinations in vision-language models. Such models sometimes claim to see objects that are not in the image. Contrastive decoding tries to fix this by building samples that provoke false objects, then pushing those outputs down.

The authors argue the method does not truly fix the problem. Gains on a common test, they say, come from two side effects. One is a crude one-way shift in the output scores. The other is a rule that in effect forces plain greedy choice. They add trick methods that score well without fixing anything. The lesson is to judge fixes by more than one benchmark score.

Catalog record

CREATORHao Yin, Gunagzong Si, Zilei Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-10020

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Mirage of Performance Gains: Why Contrastive Decoding Fails to Address Multimodal Hallucination Hao Yin, Gunagzong Si, Zilei Wang Abstract Contrastive decoding strategies are widely used to reduce hallucinations in multimodal large language models (MLLMs). These methods work by constructing contrastive samples to induce hallucinations and then suppressing them in the output distribution. However, this paper demonstrates that such approaches fail to effectively mitigate the hallucination problem. The performance improvements observed on POPE Benchmark are largely driven by two misleading factors: (1) crude, unidirectional adjustments to the model’s output distribution and (2) the adaptive plausibility constraint, which reduces the sampling strategy to greedy search. To further illustrate these issues, we introduce a series of spurious improvement methods and evaluate their performance against contrastive decoding techniques. Experimental results reveal that the observed performance gains in contrastive decoding are entirely unrelated to its intended goal of mitigating hallucinations. Our findings challenge common assumptions about the effectiveness of contrastive decoding strategies and pave the way for developing genuinely effective solutions to hallucinations in MLLMs. 1 Introduction The hallucination problem ( Li et al. 2023b ; Liu et al. 2023a ; Lovenia et al. 2023 ; Gunjal et al. 2023 ) in multimodal large language models (MLLMs) ( Bavishi et al. 2023 ; Chen et al. 2024a ; Ye et al. 2023 ; Chen et al. 2023a ) refers to the generation of outputs that are factually incorrect or misaligned with the input data. This issue arises from challenges in aligning diverse data modalities, such as text and images, which amplify reasoning errors. Such hallucinations can have serious consequences in critical domains, including autonomous driving ( Mai et al. 2023 ; Liu et al. 2023c ; Chen et al. 2023b ; Wu et al. 2023 ) (e.g., false object detection leading to accidents) and healthcare ( Wang et al. 2023 ; Hu et al. 2023 ) (e.g., incorrect diagnostic interpretations). Contrastive decoding methods ( Wu et al. 2022 ; Gupta et al. 2022 ; Niu et al. 2021 ) are widely recognized as an effective approach to addressing object hallucination in generative models. As illustrated in Figure 1 , these methods construct contrastive samples designed to induce hallucinations, then suppress the corresponding output distributions, ensuring closer alignment between model outputs and visual inputs. Representative approaches within this framework include Visual Contrastive Decoding (VCD) ( Leng et al. 2023 ) , Instruction Contrastive Decoding (ICD) ( Wang et al. 2024b ) , and Self-Introspective Decoding (SID) ( Huo et al. 2024 ) . Their training-free nature and purported ability to address hallucinations have made them highly regarded in the field. Although methods like VCD have demonstrated remarkable performance improvements on the POPE benchmark ( Li et al. 2023c ) , we reveal in Section 4 that these results are highly misleading. In reality, these methods fail to effectively address model hallucination. The observed performance gains on the POPE benchmark are primarily driven by two factors: Misleading Nature of Performance Improvement \mathcal{R}_{1} : A unidirectional adjustment of the model output distribution, which simply biases the model towards producing more ”Yes” outputs, leading to a balanced distribution on certain datasets. \mathcal{R}_{2} : The adaptive plausibility constraints in these methods degrade the sampling decoding strategy into an approximation of greedy search, resulting in deceptively improved performance. To expose the misleading nature of the performance improvement in the first scenario, we implemented two forced distribution adjustment algorithms in Section 5.1 to show that the apparent gains of contrastive decoding on the POPE Benchmark are not genuine. The methods are as follows: (1) Prompt-Based Adjustment, where we added a hint to the instruction, such as ”Whenever possible, please select Yes,” to bias outputs toward ” Yes ”; and (2) Output Layer Modification, where we altered the output layer to favor ”Yes” when the probabilities for ” Yes ” and ” No ” were similar. Although neither method mitigates hallucinations, both achieved performance gains comparable to those of contrastive decoding, confirming that these improvements do not represent a genuine solution to the problem. To highlight the misleading nature of the performance improvement in the second scenario, we incorporated the adaptive plausibility constraint into the standard sampling strategy and compared its predictions with those from contrastive decoding in Section 5.2 . The experimental results reveal that, despite having no theoretical connection to hallucination mitigation, the adaptive plausibility constraint accounts for nearly all the performance gains attributed to contrastive decoding. This finding underscores that the contrastive decoding methods, in essence, fail to mitigate hallucinations. Overall, this paper makes the following three contributions: • We identified that the performance improvement of contrastive decoding methods stems from its unidirectional and blunt adjustment of the output distribution, which coincidentally balances the distribution on certain datasets. • We discovered that another key factor driving the performance gains of contrastive decoding methods is their adaptive plausibility constraints, which streamline the sampling strategy into an approximation of greedy search. • We developed a series of spurious improvement methods and evaluated their performance against contrastive decoding methods. Our findings convincingly show that contrastive decoding methods do not alleviate hallucinations in any meaningful way. 2 Related Work 2.1 Multimodal Large Language Models The evolution of MLLMs ( Chen et al. 2024b ; Chen et al. 2024c ) has progressed from BERT-based decoders ( Li et al. 2020 ; Li et al. 2021 ) to advanced LLM architectures ( Touvron et al. 2023a ; Touvron et al. 2023b ; Meta 2024 ) , enabling more effective multimodal relationship modeling ( Chiang & Li 2023 ; Taori et al. 2023 ; Bai et al. 2023a ) . Models such as BLIP-2 ( Li et al. 2023a ) and MiniGPT-4 ( Zhu et al. 2023 ) employ Q-Former mechanisms to enhance the alignment between visual and textual inputs, facilitating more precise cross-modal interactions. InstructBLIP ( Dai & et al 2023 ) extends this framework by integrating task-specific instructions, improving the model’s ability to interpret context-sensitive visual semantics. Meanwhile, LLaVA ( Liu et al. 2023b ; Liu et al. 2024 ; Li et al. 2024 ) and Qwen-VL ( Bai et al. 2023b ) adopt simpler linear projection methods that streamline alignment, leading to superior performance in vision-language tasks. Despite these advancements, hallucination remains a persistent challenge that warrants further investigation. 2.2 Contrastive Decoding Strategies Contrastive decoding ( Yan et al. 2023 ; Zhibo et al. 2023 ; Han et al. 2022 ) are widely recognized as effective in addressing object hallucination in generative models. Visual Contrastive Decoding (VCD) ( Leng et al. 2023 ) addresses object hallucination by comparing output distributions generated from standard visual inputs and distorted visual inputs. This approach reduces the model’s dependence on linguistic priors within integrated LLMs and minimizes the impact of statistical biases in MLLM pretraining corpus. Instruction Contrastive Decoding (ICD) ( Wang et al. 2024b ) , in contrast, focuses on the role of instruction perturbations in amplifying hallucinations. By examining the differences in output distributions between standard and perturbed instructions, ICD detects hallucination-prone content and mitigates its impact effectively. Building upon these two hallucination mitigation methods, numerous approaches, including Adaptive Focal-Contrast Decoding (HALC) ( Chen et al. 2024d ) , Self-Introspective Decoding (SID) ( Huo et

Read the complete text