Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models

Marcel Gröpl ∗,, Jaewoo Jung ∗,, Seungryong Kim, Marc Pollefeys, Sunghwan Hong · 2026

Read Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models by Marcel Gröpl ∗,, Jaewoo Jung ∗,, Seungryong Kim, Marc Pollefeys, Sunghwan Hong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08456.txt.

Introduction

Marcel Gröpl and four colleagues frame visual grounding in vision language models as evidence retrieval at test time, with no extra training. Their method turns the model's own next token uncertainty into a relevance map over image tokens. It then ranks coherent regions and zooms in again until a spatial entropy rule says stop. Tests on seven benchmarks across four model designs show steady gains, largest where fine detail and high resolution matter, with clearer evidence locations.

Catalog record

CREATORMarcel Gröpl ∗,, Jaewoo Jung ∗,, Seungryong Kim, Marc Pollefeys, Sunghwan Hong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45348
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08456

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models Marcel Gröpl ∗,, Jaewoo Jung ∗,, Seungryong Kim, Marc Pollefeys, Sunghwan Hong Abstract Despite rapid progress, pretrained vision–language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in documents and compositional queries. We address this by framing grounding as test-time evidence retrieval: given a query, the model should actively identify where to look next to resolve ambiguity. To this end, we propose a training-free, model-intrinsic grounding method that uses uncertainty as supervision. Specifically, we compute the entropy of the model’s next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map, without auxiliary detectors or attention-map heuristics. We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule to avoid over-refinement. Experiments on seven benchmarks across four VLM architectures demonstrate consistent improvements over existing methods, with the largest gains on detail-critical and high-resolution settings, while also producing more interpretable evidence localizations. Figure 1 : Teaser. As shown in (a) and (b), existing VLMs struggle to answer questions when visual evidence is fine-grained or exists in spatially disjoint regions. We propose a training-free method where we apply a query-based visual grounding method to discover relevant regions and provide these regions as additional image crops, improving performance in both challenging scenarios. 1 Introduction Vision–Language Models (VLMs) have shown impressive performance on a broad range of multimodal tasks, including visual question answering and document understanding [ 21 , 4 , 31 , 6 , 36 , 7 ] . Yet many failures persist when the decisive evidence is fine-grained, e.g., small text, symbols, or scattered across disjoint regions, e.g., documents, compositional queries [ 13 , 37 ] , as exemplified in Fig. 1 . These failures suggest that the key challenge is not generation, but selective perception : identifying the right visual evidence. Accordingly, we cast such vision–language tasks as active perception at inference time: conditioned on a query, the model should perform grounding or decide where to look to acquire informative evidence. A growing body of work explores training-free ways to extract grounding signals from pretrained VLMs [ 37 , 15 ] , avoiding additional training that can be prohibitively costly for today’s large-scale models. Typically, many methods rely on attention maps for numerous downstream tasks [ 1 , 8 , 9 , 2 ] , but converting attention into reliable localization typically requires heuristic choices, e.g., selecting heads/layers and designing post-processing, and the resulting maps can be brittle across backbones and dominated by a single salient region—a poor fit for queries that require aggregating spatially disjoint evidence. A complementary line of work acts on these imperfect signals by cropping or zooming into the predicted evidence at test time. ViCrop [ 37 ] exemplifies this direction, showing that zooming can improve downstream accuracy, but its region selection remains heuristic and it typically commits to a single crop. As a result, current approaches provide limited support for principled multi-region retrieval and controlled refinement: if the initial saliency is coarse or biased toward one cue, a one-shot zoom may miss secondary evidence, whereas exhaustive grid-based zooming can recover it only at substantial computational cost. In this work, we propose a training-free, model-intrinsic grounding approach that treats inference as test-time evidence acquisition guided by the model’s own uncertainty. Our key insight is that the entropy of the next-token distribution provides a decision-relevant indicator of visual evidence. Given an image and a query, we compute the entropy of the next-token distribution at a single decoding step and backpropagate it to the visual embeddings. The resulting entropy-gradient map highlights regions whose visual features most affect the model’s uncertainty, producing a grounding signal that is tightly coupled to the model’s prediction behavior—without auxiliary detectors or attention heuristics. Crucially, our uncertainty-driven signal supports multi-region localization. We convert the entropy-gradient response into multiple spatially coherent regions of interest, enabling explicit aggregation over spatially disjoint evidence. To further recover fine-grained or secondary cues that can be suppressed in a single-pass map by dominant activations, we introduce an iterative refinement procedure that reapplies the same entropy-backpropagation mechanism on selected subregions. We regulate refinement with a spatial-entropy stopping criterion, terminating when spatial concentration no longer improves; empirically, probability-based confidence measures are inconsistent for this control, while spatial entropy provides a stable convergence signal across backbones and tasks. We evaluate our approach on multiple benchmarks [ 23 , 32 , 24 , 34 , 29 , 19 , 12 ] using strong pretrained backbones [ 20 , 21 , 31 , 5 ] . Our method consistently improves over these baselines and outperforms the strongest training-free competitor, with gains that are particularly pronounced on detail-critical and multi-evidence queries. Qualitative results further show that our refinement yields more precise and interpretable evidence localization. Our contributions are summarized as follows: • We introduce a training-free, model-intrinsic grounding method for pretrained VLMs that treats inference as test-time evidence acquisition, using the model’s own uncertainty to guide where to look. • We develop a multi-region selection and ranking pipeline that extracts multiple coherent ROIs from entropy-gradient maps, enabling retrieval and aggregation of spatially disjoint evidence required by documents and compositional queries. • We introduce an iterative refinement procedure that repeatedly re-grounds and re-crops selected regions, together with a spatial-entropy stopping criterion that provides early stopping. • We provide extensive quantitative and qualitative evaluations across multiple benchmarks and demonstrate consistent improvements over baselines. 2 Related Work Visual Grounding in MLLMs. Visual grounding aims to localize image regions relevant to a textual query. Classical approaches rely on supervised detectors, e.g., YOLO [ 25 ] , or open-vocabulary detectors that condition directly on text, e.g., Grounding DINO [ 22 ] . Only limited work studies training-free grounding directly from pretrained MLLMs. ViCrop [ 37 ] extracts localization cues from attention-derived signals across forward passes, while other analyses probe attention heads and use heuristic criteria, e.g., spatial entropy, to select informative heads [ 15 ] . Several methods instead introduce extra components to convert internal signals into masks, such as F-LMM [ 33 ] and LISA [ 17 ] . More generally, gradient-based attribution methods, e.g., Grad-CAM [ 27 ] , Integrated Gradients [ 30 ] , indicate that prediction gradients encode spatial saliency, but they are primarily designed for post-hoc interpretability. In contrast, we derive grounding from gradients of an entropy-based objective over the language model’s output distribution, yielding a training-free, model-intrinsic signal that enables principled multi-region localization without architectural modification. Resolution Constraints and Region-Level Reasoning. Reasoning over fine-grained visual evidence remains challenging for MLLMs due to limited spatial granularity and token budgets. Many systems encode the image with fixed-resolution pipelines, e.g., LLaVA-1.5 [ 20 ] , while others employ multi-crop or tiling strategies to increase visual coverage, e.g., LLaVA-1.6 [ 21 ] , InternVL [ 31 ] , Molmo [ 6 ] . Qwen-VL processes images at native resolution without predefined cropping [ 4 ] , and LLaVA-OneVision [ 3 ] explores native-resolution encoding via RiCe-ViT [ 35 , 18 ] . Beyond static resolution strategies, several works introduce explicit region-level reasoning. TEVA [ 13

Read the complete text