HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision–Language Models

Sushant Gautam, Michael A. Riegler, Pål Halvorsen [email protected]@[email protected] · 2025

Read HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision–Language Models by Sushant Gautam, Michael A. Riegler, Pål Halvorsen [email protected]@[email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12693.txt.

Introduction

Sushant Gautam, Michael A. Riegler, and Pål Halvorsen present HEDGE, a framework for detecting hallucinations in vision-language models that answer questions about images. Such models can state false things with confidence, a real risk in medical use. HEDGE probes a model by perturbing the image in controlled ways, sampling its answers, and clustering those answers by meaning. Shifts in the answer pattern then signal how shaky the model's response really is.

The team tested three medical vision-language models on two medical question datasets. Detectability depended on model design and prompt style: models with dense visual tokenization were easiest to audit, and short label-style answers gave cleaner signals than full sentences. One metric, VASE, gave the most robust signal across settings. The authors frame detection as a geometry problem shaped by sampling, prompts, and clustering choices.

Catalog record

CREATORSushant Gautam, Michael A. Riegler, Pål Halvorsen [email protected]@[email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12693

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision–Language Models Sushant Gautam, Michael A. Riegler, Pål Halvorsen [email protected]@[email protected] Abstract Vision–language models (VLMs) enable open-ended visual question answering but remain prone to hallucinations. We present HEDGE, a unified framework for hallucination detection that combines controlled visual perturbations, semantic clustering, and robust uncertainty metrics. HEDGE integrates sampling, distortion synthesis, clustering (entailment- and embedding-based), and metric computation into a reproducible pipeline applicable across multimodal architectures. Evaluations on VQA-RAD and KvasirVQA-x1 with three representative VLMs (LLaVA-Med, Med-Gemma, Qwen2.5-VL) reveal clear architecture- and prompt-dependent trends. Hallucination detectability is highest for unified-fusion models with dense visual tokenization (Qwen2.5-VL) and lowest for architectures with restricted tokenization (Med-Gemma). Embedding-based clustering often yields stronger separation when applied directly to the generated answers, whereas NLI-based clustering remains advantageous for LLaVA-Med and for longer, sentence-level responses. Across configurations, the VASE metric consistently provides the most robust hallucination signal, especially when paired with embedding clustering and a moderate sampling budget ( n ≈ 10 n{\approx}10 –15). Prompt design also matters: concise, label-style outputs offer clearer semantic structure than syntactically constrained one-sentence responses. By framing hallucination detection as a geometric robustness problem shaped jointly by sampling scale, prompt structure, model architecture, and clustering strategy, HEDGE provides a principled, compute-aware foundation for evaluating multimodal reliability. The hedge-bench PyPI library enables reproducible and extensible benchmarking, with full code and experimental resources available at https://github.com/Simula/HEDGE . 1 Introduction Large Vision-Language Models (VLMs) have rapidly advanced multimodal reasoning, enabling free-form question answering and grounded dialogue across complex visual scenes. Yet their expressive power comes with a critical weakness: hallucination—the generation of confident statements inconsistent with visual evidence or logical context. Hallucination undermines the reliability of VLMs in high-stakes domains such as clinical decision support, scientific analysis, and autonomous systems, where factual precision and interpretability are essential. 1.1 Limitations of existing metrics Recent work has sought to quantify hallucination through uncertainty and consistency metrics. Semantic Entropy (SE) Farquhar et al. 2024 measures uncertainty over semantically grouped responses via normalized mean log-likelihoods, while RadFlag Zhang et al. 2024 introduces a hallucination flagger for medical VLMs based on contradiction frequency. Vision-Amplified Semantic Entropy (VASE) Liao et al. 2025 extends SE by contrasting entropy distributions under clean versus perturbed visual inputs, offering robustness-oriented evaluation. Although these methods form a strong foundation, they share key limitations: (i) they rely on text-level token statistics, ignoring the geometry of the underlying multimodal representations; (ii) they often embed domain-specific heuristics that hinder cross-domain generalization; and (iii) they can be computationally fragile, particularly for pairwise Natural Language Inference (NLI) -based clustering. 1.2 From token uncertainty to geometric stability We posit that hallucination is not merely a probabilistic irregularity but a geometric instability within the visual-linguistic manifold. A reliable model should maintain consistent semantic neighborhoods when its visual or linguistic inputs are perturbed. Hallucination manifests when this geometric structure fractures—when embeddings that should remain close under minor perturbations disperse in latent space. This view reframes hallucination estimation as a problem of dense geometric entropy: measuring how the topology of the model’s representation space deforms under controlled variation. 1.3 Overview of the framework We introduce Hallucination Estimation via Dense Geometric Entropy (HEDGE) , a domain-agnostic framework for hallucination detection based on representation stability. HEDGE systematically probes a model’s multimodal manifold through dense visual perturbations and computes geometric entropy across semantically clustered responses. It unifies two complementary clustering strategies: (i) NLI -based clustering, which measures entailment consistency among answers Williams et al. 2018 ; He et al. 2020 ; and (ii) embedding-based clustering, which groups semantically aligned responses in representation space Reimers and Gurevych 2019 ; Gao et al. 2021 . By analyzing entropy within these clusters, HEDGE estimates hallucination likelihood without requiring task-specific labels or handcrafted thresholds. 1.4 Why medical benchmarks? Although HEDGE is not medical-specific, we evaluate it on VQA-RAD Lau et al. 2018 and KvasirVQA-x1 Gautam et al. 2025 because medical VQA offers a controlled setting for isolating hallucination behavior. These datasets provide (i) short, low-ambiguity answers that reduce linguistic variance; (ii) high-resolution images with consistent visual structure, making perturbations easier to parameterize and evaluate; and (iii) established hallucination baselines— SE Farquhar et al. 2024 , RadFlag Zhang et al. 2024 , and VASE Liao et al. 2025 —that support rigorous, reproducible comparison. We use medical imagery as a structured stress test for representation stability rather than as a domain constraint: the HEDGE pipeline is domain-agnostic and can be instantiated on other VQA settings by adopting perturbations and prompting conventions suited to the target domain. 1.5 Architectural factors in hallucination geometry Our analysis spans three representative VLMs families with distinct vision encoders and fusion mechanisms: (1) LLaVA-Med Li et al. 2023c adopts a Contrastive Language-Image Pre-training (CLIP) ViT-L/14 vision tower with images normalized to 336 px and a dual-tower alignment module; (2) Med-Gemma Sellergren et al. 2025 employs a SigLIP-based encoder operating on 896 \times 896 px crops, which are compressed into a fixed 256-token visual bottleneck; and (3) Qwen2.5-VL Bai et al. 2025 processes images at dynamically selected resolutions using a unified cross-modal transformer that preserves variable-length visual-token sequences. These architectural differences shape the local geometry of visual embeddings and thereby directly influence the detectability of hallucination under input perturbations. 1.6 Contributions In this work, we contribute two main components. First, we introduce HEDGE , a unified and domain-agnostic framework for hallucination detection in vision–language models that standardizes sampling, clustering, and robustness-based scoring. Second, we provide a systematic comparison of design choices within this framework, including NLI-based versus embedding-based clustering, answer-length prompting, and sampling scales. Our embedding-based clustering variants are new additions designed to compete with the NLI-based clustering used in prior work, and all reported results are produced within the same HEDGE pipeline to enable controlled and reproducible comparisons. 1.7 Takeaway HEDGE reframes hallucination detection from a metric-centric exercise into a study of representation stability. By quantifying geometric entropy rather than token-level uncertainty, it exposes how architectural choices, visual tokenization density, and fusion topology govern a model’s susceptibility to hallucination—establishing a principled, scalable foundation for robust VLM evaluation. 2 Related Work Hallucination detection connects to uncertainty modeling, semantic grouping, consistency-based evaluation, entailment reasoning, retrieval-grounded attribution, multimodal robustness, and medical reliability assessment. We briefly review these directions and, for each, indicate how they motivate the HEDGE framework. 2.1 Uncertainty and Semantic Dispersion Uncertainty quantification provides a core mechanism for assessing hallucination risk in both Large Language Models (LLMs) and VLMs . Early reference-free metrics such as AvgProb, MaxProb, AvgEnt, and MaxEnt estimate token-level confidence or entropy but remain sensitive to paraphrasing and lexical variation Li et al. 2024b . Semantic Entropy (SE) reduces this sensitivity by modeling dispersion across semantically grouped samples Farquhar et al. 2024 , and Semantic Entropy Probes (SEP) approximate SE with lightweight regressors over hidden states Kossen et al. 2024 . In multimodal reasoning, VASE compares semantic entropy under clean and perturbed visual inputs Liao et al. 2025 , and DSE introduces discretized semantic variability tailored to radiology filtering Wienholt et al. 2025 . More broadly, perturbation-based UQ methods apply paraphrasing, visual occlusion, or structured noise to quantify variability Gao et al. 2024 ; Padhi et al. 2025 ; Wang and Ji 2024 , but may suffer from calibration

Read the complete text