GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations

Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique · 2026

Read GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations by Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10978.txt.

Introduction

Boyuan Chen and coauthors tackle a stubborn weakness of vision-language models. They hallucinate counts. They score far worse at counting objects than at other visual reasoning. Their GroundCount framework feeds the language model explicit spatial grounding. That grounding comes from a conventional object detector through structured prompts. It does not come through fusing features. Prompt-based grounding beats feature-level fusion in their ablations. Positional encoding proves critical. Confidence scores mostly add noise. One model's reflective style clashes with the structured prompts, though.

Catalog record

CREATORBoyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35604
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10978

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations Boyuan Chen, Minghao Shao, Siddharth Garg, Ramesh Karri, Muhammad Shafique Abstract Vision Language Models (VLMs) exhibit persistent hallucinations in counting tasks, with accuracy substantially lower than other visual reasoning tasks (excluding sentiment). This phenomenon persists even in state-of-the-art reasoning-capable VLMs. Conversely, CNN-based object detection models (ODMs) such as YOLO excel at spatial localization and instance counting with minimal computational overhead. We propose GroundCount , a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. In the best case, our prompt-based augmentation strategy achieves 81.3% counting accuracy on the best-performing model (Ovis2.5-2B) - a 6.6pp improvement - while reducing inference time by 22% through elimination of hallucination-driven reasoning loops for stronger models. We conduct comprehensive ablation studies demonstrating that positional encoding is a critical component, being beneficial for stronger models but detrimental for weaker ones. Confidence scores, by contrast, introduce noise for most architectures and their removal improves performance in four of five evaluated models. We further evaluate feature-level fusion architectures, finding that explicit symbolic grounding via structured prompts outperforms implicit feature fusion despite sophisticated cross-attention mechanisms. Our approach yields consistent improvements across four of five evaluated VLM architectures (6.2–7.5pp), with one architecture exhibiting degraded performance due to incompatibility between its iterative reflection mechanisms and structured prompts. These results suggest that counting failures stem from fundamental spatial-semantic integration limitations rather than architecture-specific deficiencies, while highlighting the importance of architectural compatibility in augmentation strategies. I Introduction Vision-Language Models (VLMs) have demonstrated remarkable progress across a wide range of multimodal tasks [ 34 ] , benefiting from large-scale training and fine-tuning on massive image-text corpora. This joint learning paradigm significantly enhances both contextual reasoning and instruction-following abilities [ 22 ] . Despite such advances, VLMs exhibit hallucinations by generating object attributes, entities, or spatial relations inconsistent with the visual input [ 47 , 49 ] . A growing body of work attributes this issue to an imbalanced cross-modal attention mechanism: the language decoder often over-attends to textual priors while under-utilizing visual tokens [ 15 , 40 , 7 ] . Consequently, several methods have been proposed to mitigate hallucination, either through decoding-level adjustments [ 42 , 46 , 13 ] or training-level regularization [ 41 , 44 ] . Motivated by layer-level analysis showing that hallucination primarily occurs when later layers deviate from correct logits, another direction proposes steering strategies [ 19 , 38 , 37 ] that preserve the correct momentum from earlier layers. These approaches have yielded measurable improvements on popular VLMs, including LLaVA-v1.5 [ 21 ] , LLaVA-v1.6 [ 22 ] , InstructBLIP [ 9 ] , MiniGPT-2 [ 5 ] , and mPLUG-Owl2 [ 45 ] . Recent work jointly trains vision encoders and language decoders on diverse multimodal datasets [ 48 ] , and incorporates reinforcement learning to enhance reasoning capabilities [ 43 , 39 , 26 ] . These efforts have led to significantly stronger multimodal representations and improved factual alignment. However, even state-of-the-art (sota) VLMs continue to exhibit systematic hallucination in counting tasks. As we demonstrate in Section III and Table I , counting consistently remains the lowest-accuracy task across all evaluated VLMs (excluding sentiment, which is relatively subjective), with accuracies ranging from 64.0% to 74.7%. This is substantially lower than other visual reasoning tasks such as object recognition (70.3% to 89.6%) and attribute identification (83.2% to 86.5%). Recent research also supports this finding [ 36 , 11 ] . The persistence of counting hallucinations in reasoning-capable VLMs presents a unique challenge. Unlike earlier VLMs where attention steering and decoding adjustments proved effective, modern reasoning VLMs already incorporate reflection mechanisms [ 26 , 48 , 39 ] that direct attention to visual modalities. Furthermore, the iterative reasoning process makes it difficult to apply layer-level vector steering, as there is no single ”correct” token to steer toward during multi-step reasoning. This suggests that existing hallucination mitigation strategies, while effective for simpler VLMs, are insufficient for addressing systematic failures in compositional tasks like counting. On the other hand, counting the number of occurrences of an object is nearly trivial for object detection models (ODMs), such as YOLO [ 14 ] and DETR [ 4 ] . These models achieve very high accuracy in object detection while requiring minimal inference time compared to auto-regressive VLMs. Moreover, ODMs provide structured outputs including bounding boxes, confidence scores, and class labels. This information explicitly addresses the spatial and compositional reasoning needed for accurate counting. This observation suggests a complementary approach: rather than attempting to fix VLM attention mechanisms or reasoning processes, we can augment VLMs with explicit grounding information from specialized object detection models. Our contributions are as follows: • We provide a comprehensive analysis of sota VLMs in counting tasks, demonstrating that counting remains systematically the lowest-accuracy task across all evaluated models despite advances in reasoning capabilities. • We introduce GroundCount , an object-detection-driven augmentation pipeline that increases VLM counting accuracy by 6.2 to 7.5pp across four of five evaluated models (up to 81.3% on the PhD benchmark) with negligible memory overhead and reduced inference time for stronger models (Ovis2.5-2B: -22%, Molmo2-4B: -23%). • We conduct three groups of ablation studies analyzing the impact of different ODM outputs (confidence scores, positional encoding, detection thresholds) on VLM performance, providing insights into why grounding information reduces hallucination. • We evaluate a fusion architecture that combines VLMs with ODMs at the feature level through fine-tuning, analyzing the trade-offs between prompt-based augmentation and architectural integration. TABLE I: Correctness rates (%) of different VLMs on the PhD benchmark using greedy decoding, with explicit thinking mode enabled when available (i.e., the output begins with a prior reasoning segment enclosed by special tokens such as "<think>...</think>" ). The PhD benchmark consists of questions from five task types listed in the Task column on the left. For each subset (excluding sentiment), the task with the lowest correctness rate is highlighted with a light-red background. Results across all four evaluation settings are shown; baseline corresponds to standard VQA without adversarial context. VLM Molmo2-4B Ovis2.5-2B R-4B Qwen3-VL-2B InternVL3.5-1B Task base sec icc ccs base sec icc ccs base sec icc ccs base sec icc ccs base sec icc ccs object 89.6 79.2 65.9 89.2 88.5 85.1 84.0 84.6 87.9 85.3 82.1 90.4 87.7 84.3 83.1 83.1 70.3 68.3 67.9 82.0 attribute 85.2 44.7 15.1 93.9 86.5 76.8 75.3 89.2 86.5 75.5 64.5 92.7 86.5 77.8 74.2 88.1 83.2 75.2 73.6 80.9 positional 80.8 66.4 47.3 89.1 80.9 73.0 65.3 88.6 80.6 71.5 54.5 94.1 78.3 71.3 63.6 86.8 70.2 61.3 49.2 85.9 counting 67.4 62.9 33.1 87.9 74.7 67.6 52.0 79.8 73.7 69.4 44.2 87.9 69.4 63.5 53.3 83.1 64.0 54.0 41.3 85.5 sentiment 67.3 52.9 32.0 93.6 68.4 59.6 59.2 94.9 68.1 57.9 50.0 96.2 68.1 50.9 50.3 91.0 66.2 57.7 59.7 83.3 II Related Works VLM Hallucination in Counting Tasks : Vision-Language Models exhibit persistent hallucinations in counting tasks despite advances in reasoning capabilities. One major hypothesis attributes this to vision transformers (ViTs) having difficulties noticing fine-grained image details [ 28 , 30 ] , as ViTs process images through global patch attention rather than the local receptive fields that facilitate spatial instance

Read the complete text