CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models
Neeraj Anand, Samyak Jha, Udbhav Bamba, Rahul Rahaman · 2026
Read CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models by Neeraj Anand, Samyak Jha, Udbhav Bamba, Rahul Rahaman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00659.txt.
Introduction
Neeraj Anand, Samyak Jha, Udbhav Bamba, and Rahul Rahaman present CRoPS, a training-free method for stopping vision-language models from hallucinating content that is not in an image. Existing fixes often assume one narrow cause of hallucination, and their protection fades near the end of a long answer, exactly where invented details tend to creep in.
A common trick contrasts the real model with a version blinded to the image, but the authors show visual information still leaks into the text that way. Their hallucinated comparison model instead removes key text tokens selectively. A generalized contrastive decoding step then steers generation away from that hallucinated behavior throughout the answer, aiming for reliability without any retraining.
Catalog record
| CREATOR | Neeraj Anand, Samyak Jha, Udbhav Bamba, Rahul Rahaman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00659
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models Neeraj Anand, Samyak Jha, Udbhav Bamba, Rahul Rahaman Abstract Despite the rapid success of Large Vision-Language Models (LVLMs), a persistent challenge is their tendency to generate hallucinated content, undermining reliability in real-world use. Existing training-free methods address hallucinations but face two limitations: (i) they rely on narrow assumptions about hallucination sources, and (ii) their effectiveness declines toward the end of generation, where hallucinations are most likely to occur. A common strategy is to build hallucinated models by completely or partially removing visual tokens and contrasting them with the original model. Yet, this alone proves insufficient, since visual information still propagates into generated text. Building on this insight, we propose a novel hallucinated model that captures hallucination effects by selectively removing key text tokens. We further introduce Generalized Contrastive Decoding , which integrates multiple hallucinated models to represent diverse hallucination sources. Together, these ideas form CRoPS , a training-free hallucination mitigation framework that improves CHAIR scores by 20% and achieves consistent gains across six benchmarks and three LVLM families, outperforming state-of-the-art training-free methods. 1 1 1 footnotetext: Code is available at https://github.com/ubamba98/CRoPS-Mitigate-Hallucinations-in-Vision-Language-Models 1 Introduction Recent advances in large vision-language models (LVLMs) have demonstrated remarkable multi-modal capabilities ( 26 ; 4 ) . Their ability to comprehend both images and language has enabled a wide range of applications. However, similar to Large Language Models (LLMs), LVLMs are prone to “hallucinations” generating convincing responses that lack precision, which can lead to misleading information ( 30 ; 18 ) . This limitation poses a significant challenge to their reliability as trustworthy AI assistants in real-world applications ( 35 ; 23 ) . Recent works aiming to mitigate hallucinations can be categorized into two regimes: training-based methods ( 1 ; 12 ; 31 ) , and training-free methods ( 37 ; 19 ; 17 ; 36 ; 5 ) . In this study, we are interested in a very specific group of training-free approaches that utilize a common framework of Contrastive Decoding (CD) ( 19 ) . CD, a training-free approach, proposes to negate the hallucinations from LLM outputs by contrasting (subtracting) outputs of heavily hallucinated models, essentially reducing the problem to identifying and using hallucinated models to contrast with. These methods often suffer from several shortcomings. Firstly, by design, the hallucinated models used for contrasting cease to perform well as the generation of output tokens (Figure 3 ), making them effective primarily in the early stages of generation.Furthermore, a single hallucinated model does not sufficiently represent all the sources of hallucination, e.g., hallucination due to uninformative visual tokens and hallucinations from the bias produced by the training data. In this work, we start with an analysis of the existing methods ( 5 ; 10 ) and show, by means of both empirical analysis and theoretical computations, how existing methods struggle to remove hallucination as the LVLM generates more tokens. We then propose a novel hallucinated model that overcomes this challenge by going beyond the removal of full or partial visual tokens and considering textual inputs for removal. Our proposed model is motivated by what our analysis shows, that in the later stages of the generation, previously generated output tokens carry equal, if not more, importance than visual tokens. Next, we empirically show how existing contrastive decoding methods fail to address different kinds of sources of hallucination. To mitigate this issue, we generalize the concept of contrastive decoding and extend it to accommodate multiple models to contrast with rather than a single model. Our proposed Generalized Contrastive Decoding , allows us to conjoin our newly proposed hallucinated model with existing works 10 , to devise our novel, training-free hallucination mitigation method CRoPS ( fig. 1 ). We evaluate and compare our proposed CRoPS against competing methods in a wide range of generative and discriminative benchmarks. We also show that CRoPS outperforms all methods across three different choice of architectures. Contribution: Below we summarize our main contributions. In this work, • We provide detailed analysis of how existing methods perform poorly in the later stages of the LVLM token generation. This also enables us to devise a novel hallucinated model by moving beyond visual tokens to identify and remove textual inputs from prompt and past generated tokens. • We empirically show how the individual methods only tackle a specific source of hallucination and fail to remove other sources. To this end, we formulate Generalized Contrastive Decoding, by extending the scope of contrastive decoding to allow multiple models to be contrasted with. • Finally we combine our novel text-deficit hallucinated model with image-deficit hallucinated model under the novel generalized contrastive decoding framework and propose CRoPS. Our proposed method significantly outperforms baseline, and brings consistent improvement over competing methods across a wide range of tasks, datasets, and LVLM architectures. Figure 1 : Overview of CRoPS framework. CRoPS combines two hallucinated models: one that removes visual tokens to capture vision-related hallucinations, and another that removes key textual tokens to capture text-related hallucinations. Their outputs are then integrated through generalized contrastive decoding framework to reduce hallucinations in LVLMs. 2 Related Works With the progress in LLMs, recent studies have explored LVLMs by integrating visual encoders into pre-trained LLMs. These models have shown some advanced multi-modal capabilities. However, they suffer from hallucinations ( 31 ; 45 ; 7 ; 40 ) , which restrict their real-world applications ( 35 ; 23 ) . There are several causes of hallucinations, including a lack of understanding of world knowledge, overfitting to specific training data patterns, and insufficient common sense reasoning. In LLMs, hallucinations typically occur when generated responses contradict real-world knowledge or common sense. In contrast, for VLMs, the main concern is whether the generated response conflicts with the provided image. Researchers have explored several methods to mitigate this issue, which can be broadly categorized into the following two groups: Training-based Approaches. Training-based methods involve either retraining VLMs with curated datasets or using auxiliary models to supervise or revise generations. These approaches include instruction fine-tuning on hallucination-aware datasets ( 16 ; 8 ; 47 ; 11 ; 43 ; 44 ) and post-hoc training of revisor networks that analyze the model outputs and correct hallucinations using auxiliary networks ( 29 ; 49 ; 42 ; 2 ; 39 ; 6 ) . While these techniques can be effective, they require extensive computational resources and careful dataset design. Training-free Approaches. Training-free approaches, in contrast, modify inference-time decoding or attention patterns without additional training or supervision. They include decoding-time modifications ( 10 ; 35 ; 5 ; 19 ; 3 ; 27 ; 17 ; 36 ; 14 ; 50 ; 9 ; 13 ; 38 ; 48 ; 20 ) and attention adjustment mechanisms ( 32 ; 41 ) . These methods are simple plug-and-play modules and avoid the computational overhead of training-based methods. Closely related to our work are Multi-Modal Mutual-Information Decoding (M3ID) ( 5 ) and Self Introspective Decoding (SID) ( 10 ) , which also fall under the category of training-free approaches. M3ID proposes to utilize and tweak the same LLM to come up with a hallucinated model. They argue that if the image input is removed from the LLM, the model