To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models
OFM Riaz Rahman Aranya, Kevin Desai · 2026
Read To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models by OFM Riaz Rahman Aranya, Kevin Desai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22623.txt.
Introduction
Aranya and Desai test medical vision-language models for two linked flaws. One flaw is hallucination, where a model invents findings. The other is sycophancy, where it folds under user pressure. They test six models on three medical question sets with images. A clear tradeoff appears: models that hallucinate least tend to agree most under pressure. The authors propose three scores to capture grounding, pressure resistance, and safety. Across 1,151 cases, no model scores above 0.35 on their safety index. They argue both flaws must be checked before clinical use.
Catalog record
| CREATOR | OFM Riaz Rahman Aranya, Kevin Desai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34784 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22623
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models OFM Riaz Rahman Aranya, Kevin Desai Abstract Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA datasets and uncover a grounding-sycophancy tradeoff: models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and Clinical Safety Index (CSI), a unified safety index that combines grounding, autonomy, and calibration via a geometric mean. Across 1,151 test cases, no model achieves a CSI above 0.35, indicating that none of the evaluated 7–8B parameter VLMs is simultaneously well-grounded and robust to social pressure. Our findings suggest that joint evaluation of both properties is necessary before these models can be considered for clinical use. Code is available at https://github.com/UTSA-VIRLab/AgreeOrRight 1 Introduction Large vision-language models (VLMs) have demonstrated strong capabilities on natural image and text tasks, and recent efforts have begun adapting them to the biomedical domain 14 ; 22 . When evaluated on standard medical visual question answering (VQA) datasets, including VQA-RAD 12 , SLAKE 18 , and PathVQA 10 , several VLMs now approach or surpass prior supervised baselines, fueling interest in their potential as diagnostic assistants. Yet while these numerical gains are encouraging, they offer an incomplete picture of whether a model is suitable for clinical use. A model that correctly identifies pneumonia on a chest X-ray, for instance, provides little practical value if it reverses that finding the moment a user states that “a senior radiologist disagrees.” Figure 1 : The grounding–sycophancy tradeoff on VQA-RAD ( n{=}451 ). Each point represents one model; the x-axis shows L-VASE (hallucination propensity, lower is better) and the y-axis shows CCS (confidence-calibrated sycophancy, lower is better). No model reaches the lower-left desired quadrant. Models that hallucinate less (Qwen3-VL, MedGemma) are the most sycophantic, while the most resistant model (IDEFICS2) hallucinates substantially. Two fundamental failure modes underlie this concern. The first is hallucination : VLMs inherit from their language-model backbones a tendency to generate fluent but factually unsupported outputs 8 ; 16 . In the medical setting, this manifests as clinically plausible descriptions of findings, such as masses, effusions, or incorrect laterality, that are not present in the input image. Recent evaluations have shown that medical VLMs fine-tuned on domain data, despite achieving strong benchmark accuracy, can be more susceptible to hallucination than their general-domain counterparts 8 , raising significant concerns about domain-specific reliability. The second failure mode is sycophancy : when presented with authoritative but incorrect user assertions, VLMs tend to abandon previously correct answers in favor of alignment with the user’s stated opinion 26 ; 25 . This behavior has been documented in text-only LLMs 3 and, more recently, in vision-language settings where sycophancy rates can exceed 90% for certain model families 15 . In clinical contexts, where physician queries may carry implicit authority, sycophantic capitulation undermines the fundamental purpose of an automated decision-support tool. Recent work has shown that LLMs can be induced to confirm fabricated clinical details in up to 83% of cases 23 , further underscoring this risk. Both failure modes have received substantial individual attention. Hallucination benchmarks and detection methods have proliferated rapidly 8 ; 4 , as have studies characterizing sycophantic behavior in LLMs across general and medical domains 25 ; 3 ; 7 . However, to our knowledge, no prior work has examined these two properties jointly in medical VLMs. This is a critical gap, because the two failure modes interact in ways that have direct clinical implications: is a model that hallucinates less also more resistant to social pressure, or are these properties in tension? Our findings reveal a consistent and concerning pattern. Across six VLMs, three general-purpose and three medical-specialist, evaluated on three medical VQA benchmarks, grounding and sycophancy are anti-correlated. The models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than every medical-specialist model in our evaluation. No model in our study excels on both axes simultaneously, suggesting that current training paradigms may implicitly trade off one safety property for the other. To quantify these properties and their interaction, we introduce three metrics: 1. L-VASE (Logit-Level Visual Assertion Semantic Entropy): A hallucination metric that addresses the double-normalization issue in the original VASE formulation 17 . VASE applies softmax to contrastive differences of probability vectors, introducing a double-normalization with no principled interpretation in log-probability space. L-VASE instead operates on raw logits, where linear combinations are mathematically natural and a single softmax produces coherent entropy estimates. 2. CCS (Confidence-Calibrated Sycophancy): A sycophancy metric that weights each capitulation by the model’s own logit-derived confidence. A model that abandons a high-confidence diagnosis under social pressure represents a graver safety failure than one that changes an uncertain answer; CCS captures this distinction. 3. CSI (Clinical Safety Index): A unified safety score inspired by Failure Mode and Effects Analysis (FMEA), a widely adopted risk-assessment methodology for medical devices 20 . CSI combines grounding, autonomy, and calibration via a geometric mean, enforcing the principle that failure on any single axis renders a system clinically unsafe. 2 Related Work Medical vision-language models. The adaptation of general-purpose VLMs to the medical domain has accelerated considerably in recent years. LLaVA-Med 14 introduced a curriculum learning approach to fine-tune a general-domain VLM on biomedical figure-caption pairs, achieving competitive performance on VQA-RAD 12 , SLAKE 18 , and PathVQA 10 . Med-Flamingo 22 extended few-shot multimodal learning to the clinical setting, while more recent models such as MedVLM-R1 24 and MedGemma 27 have incorporated reasoning incentives and larger-scale medical pretraining. On the general-purpose side, LLaVA-1.5 19 , Qwen3-VL 2 , and IDEFICS2 13 have shown strong zero-shot transfer to medical benchmarks without domain-specific tuning. Despite these advances, evaluation has focused almost exclusively on accuracy, with limited attention to behavioral safety properties such as hallucination and sycophancy. Hallucination in vision-language models. Hallucination, the generation of outputs that are fluent but unsupported by the input, is a well-documented failure mode in both text-only LLMs 11 and multimodal models 16 . In the general VLM literature, POPE 16 introduced a polling-based protocol for evaluating object hallucination, while subsequent work has proposed contrastive decoding and visual grounding strategies as mitigation techniques. In the medical domain, MedVH 8 provided the first systematic benchmark for hallucination in medical VLMs, revealing that domain fine-tuned models can be more susceptible to hallucination than their general-domain counterparts. Med-HallMark 4 further introduced hierarchical hallucination categorization and a severity-aware scoring metric. For hallucination detection , Liao et al. 17 proposed VASE, which amplifies the influence of visual input by contrasting semantic distributions under clean and perturbed images. Our L-VASE builds directly on VASE but addresses a double-normalization issue in its formulation: VASE applies softmax to contrastive differences of probability vectors, which lacks a principled interpretation. L-VASE instead operates on raw logits, where linear