VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs

Govinda Kolli 1,2, Adinath Madhavrao Dukre 1, Behzad Bozorgtabar 3, Dwarikanath Mahapatra 4, Imran Razzak 1 · 2026

Read VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs by Govinda Kolli 1,2, Adinath Madhavrao Dukre 1, Behzad Bozorgtabar 3, Dwarikanath Mahapatra 4, Imran Razzak 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20314.txt.

Introduction

Kolli and co-authors tackle hallucination in medical vision-language models. Such models often answer from language habits rather than the image, a clinical risk. Their VGS-Decoding method needs no retraining and works at inference time. It scores each token by how much its odds drop when the image is degraded. Decoding then boosts grounded tokens and suppresses the rest. Tests on two medical question sets showed gains up to 9.12%. The method adds only twofold inference cost.

Catalog record

CREATORGovinda Kolli 1,2, Adinath Madhavrao Dukre 1, Behzad Bozorgtabar 3, Dwarikanath Mahapatra 4, Imran Razzak 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19565
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20314

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VGS-Decoding: Visual Grounding Score Guided Decoding for Hallucination Mitigation in Medical VLMs Govinda Kolli 1,2, Adinath Madhavrao Dukre 1, Behzad Bozorgtabar 3, Dwarikanath Mahapatra 4, Imran Razzak 1 Abstract Medical Vision-Language Models (VLMs) often hallucinate by generating responses based on language priors rather than visual evidence, posing risks in clinical applications. We propose Visual Grounding Score Guided Decoding (VGS-Decoding), a training-free method to mitigate hallucinations during inference. Our key insight is that hallucinated tokens maintain or increase their probability when visual information is degraded, while visually grounded tokens decrease in probability. We introduce the Visual Grounding Score (VGS), which measures each token’s visual dependency by comparing distributions from original and distorted images. During decoding, we reweight probabilities by amplifying visually grounded tokens while suppressing hallucinations. Unlike fixed-weight contrastive methods, VGS-Decoding provides per-token adaptive control. Experiments on MIMIC-Diff-VQA and VQA-RAD across LLaVA-Med, CheXagent, and MedGemma demonstrate consistent improvements, with up to + 9.12 % +9.12\% overall gain and + 8.98 % +8.98\% in open-ended recall, while introducing only 2 × 2\times inference overhead and no additional training, making it practical for clinical deployment. Upon acceptance, code will be released publicly to facilitate reproducibility. 1 Introduction Medical Vision-Language Models (VLMs) have emerged as powerful tools for clinical applications, demonstrating remarkable capabilities in tasks such as visual question answering (VQA), radiology report generation, and diagnostic assistance [ 16 , 3 , 24 ] . Models like LLaVA-Med [ 16 ] , trained on large-scale biomedical image-text pairs, have shown promising performance on medical VQA benchmarks including VQA-RAD [ 14 ] and MIMIC-Diff-VQA [ 11 ] . Despite these advances, medical VLMs remain vulnerable to hallucinations, generating clinically plausible but factually incorrect responses that are unsupported by the visual evidence [ 25 , 9 ] . In high-stakes medical scenarios, such hallucinations pose significant risks, potentially leading to misdiagnosis and inappropriate clinical decisions, and dangerous automation bias [ 26 ] . Hallucinations in VLMs primarily arise from over-reliance on language priors rather than visual information [ 8 , 15 , 12 ] , where models draw upon statistical patterns learned during training. This is particularly problematic in medical VQA, where precise identification of anatomical locations, pathological findings, and diagnostic details is critical. Several decoding-time strategies have been proposed to mitigate hallucinations in general-domain VLMs, building upon foundational contrastive decoding techniques in NLP [ 17 ] . Visual Contrastive Decoding (VCD) [ 15 ] contrasts output distributions from original and noise-distorted images using a subtractive formula in log-space. Decoding by Contrasting Layers(DoLA) [ 4 ] exploits the hierarchical encoding of factual knowledge in transformer layers by contrasting early and late layer outputs. Over-trust Penalty and a Retrospection-Allocation(OPERA) [ 12 ] introduces a penalty term during beam search to address over-trust in summary tokens. While these methods have shown effectiveness on general VQA benchmarks like POPE [ 18 ] and MME [ 7 ] , we find that they often fail on medical VQA, sometimes degrading performance by over 6% compared to baseline greedy decoding. This failure stems from the unique characteristics of medical VQA: shorter responses, specialized vocabulary, and the critical importance of precise anatomical and pathological terms where aggressive correction can be harmful. In this paper, we propose Visual Grounding Score Guided Decoding (VGS-Decoding) , a training-free method specifically designed for hallucination mitigation in medical VQA. Our key insight is that hallucinated tokens exhibit a distinctive signature: when visual information is degraded, hallucinated tokens maintain or increase their probability (being driven by language priors), while visually grounded tokens decrease in probability. We formalize this observation through the Visual Grounding Score (VGS), a novel metric that quantifies each token’s visual dependency Unlike VCD’s fixed-weight subtractive approach, VGS provides a bounded, interpretable measure in [-1,+1] that enables per-token adaptive correction through multiplicative reweighting. This formulation offers several advantages: (1) tokens with high VGS (visually grounded) are amplified, (2) tokens with negative VGS (potential hallucinations) are suppressed, and (3) neutral tokens remain largely unchanged, preserving the model’s linguistic coherence. We evaluate VGS-Decoding on two medical VQA benchmarks (VQA-RAD [ 14 ] and MIMIC-Diff-VQA [ 11 ] ) across three diverse medical VLMs (LLaVA-Med, CheXagent [ 3 ] , and MedGemma [ 24 ] ). Our experiments reveal that VGS-Decoding is the only method that consistently improves all models, achieving up to +9.12% overall improvement. In contrast, existing methods (VCD, DoLA, OPERA) frequently degrade performance on medical VQA, highlighting the need for domain-specific solutions. Our main contributions are: \bullet We identify that existing hallucination mitigation methods often fail on medical VQA, motivating the need for domain-adapted approaches. \bullet We propose the Visual Grounding Score (VGS), a novel metric that quantifies token-level visual dependency through normalized probability difference. \bullet We introduce VGS-Decoding, a training-free method that achieves consistent improvements across diverse medical VLMs and datasets. 2 Method We present Visual Grounding Score Guided Decoding (VGS-Decoding), a training-free approach for mitigating hallucinations in medical VLMs. Our method operates during inference by adaptively reweighting token probabilities based on their visual grounding score. Fig 1 illustrates the overall framework. Figure 1: Overview of VGS-Decoding. Both the original image V and its noise-distorted version V^{\prime} are passed through the medical VLM, producing token distributions P_{\text{orig}} (blue) and P_{\text{dist}} (green). The Visual Grounding Score (VGS) is computed as their normalized probability difference and used for multiplicative reweighting: visually grounded tokens (positive VGS) are amplified, while hallucinated tokens (negative VGS, \times ) are suppressed. 2.1 Preliminaries Given a medical image V and a text query Q , a vision-language model generates a response token-by-token in an autoregressive manner. At each decoding step t , the model produces a probability distribution over the vocabulary: P_{orig}(y_{t}|V,Q,y_{<t})=\text{softmax}(z_{t}) (1) where z_{t} denotes the logits and y_{<t} represents previously generated tokens. Standard greedy decoding selects y_{t}=\arg\max_{y}P_{orig}(y) . Hallucinations occur when the model generates tokens based on language priors rather than visual evidence [ 23 , 6 ] . For instance, when asked about the location of a finding in a chest X-ray, a model might generate “left” based on statistical co-occurrence patterns even when the image shows a right-sided abnormality. 2.2 Visual Grounding Score Our method is motivated by a key observation: visually grounded tokens and hallucinated tokens exhibit distinct behaviors when visual information is degraded. Specifically, visually grounded tokens show decreased probability when the image is corrupted, as the model loses access to the supporting visual evidence. In contrast, hallucinated tokens maintain or even increase their probability under visual corruption, since these tokens are driven by language priors rather than actual visual content. To exploit this behavioral signature, we first construct a distorted version of the input image V^{\prime}=\mathcal{D}(V) by applying a combination of Gaussian and Poisson noise [ 10 , 19 , 2 ] : V^{\prime}=V+\mathcal{N}(0,\sigma^{2})+\text{Poisson}(\lambda) (2) where \sigma and \lambda control the noise intensity. We then obtain the probability distribution conditioned on the distorted image: P_{\text{dist}}(y_{t}\mid V^{\prime},Q,y_{<t})=\text{softmax}(\mathbf{z}^{\prime}_{t}) (3) We define the Visual Grounding Score (VGS) as the normalized difference between the original and distorted distributions: \text{VGS}(t)=\frac{P_{\text{orig}}(t)-P_{\text{dist}}(t)}{P_{\text{orig}}(t)+P_{\text{dist}}(t)} (4) The VGS metric offers several desirable properties. First, it is bounded within [-1,+1] , ensuring stable and interpretable values throughout the decoding process. A positive VGS indicates that the token is visually grounded, as its probability decreases when visual information

Read the complete text