Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions
Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu Donald E. BrownSchool of Data Science, University of VirginiaCharlottesville, Virginia, USA {ss4yd, nm4wu, mcu2xn, deb}@virginia.edu · 2025
Read Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions by Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu Donald E. BrownSchool of Data Science, University of VirginiaCharlottesville, Virginia, USA {ss4yd, nm4wu, mcu2xn, deb}@virginia.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17722.txt.
Introduction
Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu, and Donald Brown probe a basic skill of vision language models: counting objects in an image. Models may lean on biases from training instead of truly looking, a flaw that shows up when a question demands careful attention to one part of a scene.
Using made up images, the team varies object number, color, texture, and wording, then tracks both accuracy and where models look. Accuracy falls as scenes and prompts grow more complex, much as human counting strains under load. Shifting attention inside the model gives small but real gains.
Catalog record
| CREATOR | Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu Donald E. BrownSchool of Data Science, University of VirginiaCharlottesville, Virginia, USA {ss4yd, nm4wu, mcu2xn, deb}@virginia.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17722
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions Saurav Sengupta, Nazanin Moradinasab, Jiebei Liu Donald E. BrownSchool of Data Science, University of VirginiaCharlottesville, Virginia, USA {ss4yd, nm4wu, mcu2xn, deb}@virginia.edu Abstract Recent research suggests that Vision Language Models (VLMs) often rely on inherent biases learned during training when responding to queries about visual properties of images. These biases are exacerbated when VLMs are asked highly specific questions that require them to focus on particular areas of the image in tasks such as counting. We build upon this research by developing a synthetic benchmark dataset and evaluation framework to systematically determine how counting performance varies as image and prompt properties change. Using open-source VLMs, we then analyze how attention allocation fluctuates with varying input parameters (e.g. number of objects in the image, objects color, background color, objects texture, background texture, and prompt specificity). We further implement attention-based interventions to modulate focus on visual tokens at different layers and evaluate their impact on counting performance across a range of visual conditions. Our experiments reveal that while VLM counting performance remains challenging, especially under high visual or linguistic complexity, certain attention interventions can lead to modest gains in counting performance. 1 Introduction While specialized object counting methods consistently outperform Vision Language Models (VLMs) on controlled benchmarks—including SAM-based detection frameworks like PseCo that leverage foundation models for class-agnostic counting 11 , multi-modal open-world counters such as CountGD that achieve state-of-the-art accuracy through grounding and visual exemplars 2 , and density estimation approaches like CrowdDiff that employ diffusion models for high-fidelity crowd density prediction 17 , robust counting capabilities can be essential for VLMs. Unlike task-specific counters that require category-specific training or few-shot examples, VLMs must support compositional queries where numerosity estimation is embedded within complex reasoning chains. From both cognitive and architectural perspectives, accurate understanding of numbers of objects in an image can be a foundational capability for general visual intelligence that can help improve downstream tasks in robotics, medical imaging and more 19 ; 18 . This work provides a systematic quantification of current open-source VLM performance in object counting, and investigates the critical role attention plays in their success or failure. Research on VLM counting performance from Guo et al . and Aghisi et al . 8 ; 1 show that VLMs are not proficient in counting. Guo et al . use 3 ‘levels’ of images and prompts with each succeeding ‘level’ being in increasing order of complexity. They measure accuracy and relative error when compared to the ground truth count. Aghisi et al . 1 evaluate the use of reasoning in improving counting performance and probe the language model for which layers contribute most to getting a more accurate count. In parallel with these efforts, we create a set of synthetic images designed to evaluate VLM counting performance. Our work differs from existing work not by merely expanding the evaluation criteria, but by providing a fine-grained diagnostic framework. Our approach provides a comprehensive analysis across four key dimensions: (1) prompt specificity, (2) visual complexity via variations in texture and color across backgrounds and objects , (3) systematic count ranges from 0-50 in increments of 10, and (4) attention distribution over vision tokens. Crucially, this framework allows us to correlate specific visual and linguistic inputs (1-3) with their mechanistic effects on model internals (4). We evaluate three open-source VLMs that perform strongly on established VLM benchmarks: 1. Qwen2.5-VL 21 2. Kimi-VL-A3B-Instruct 20 and 3. InternVL3-9B 5 . Their open-source nature enables detailed introspection into model internals. This multifaceted evaluation framework enables a more precise, mechanistic understanding of VLM counting capabilities and failure modes. We use a subset of this data to investigate how modifying attention over vision tokens changes VLM counting performance over our evaluation dataset. In summary, in this paper we: 1. Create a synthetic counting dataset that allows us to evaluate VLMs counting performance. The images, while unchallenging for modern computer vision, reveal different failure modes of VLM counting performance. They allow us to isolate specific characteristics of the image and control for other variables to investigate which part of the input prompt and input image might have contributed to the failure mode. 2. Characterize the chosen VLM’s counting performance along a) increasing prompt specificity, b) object shapes and colors, c) background colors and textures, and d) distribution of attention over vision tokens. 3. Building upon our analysis, we investigate attention modification over visual tokens, specifically in the language decoder part of the VLM. We move beyond passive observation to evaluate if actively modifying the attention over vision tokens causally contributes to increased counting accuracy. 2 Related Work Recent research has revealed significant limitations in Vision Language Models’ counting capabilities. Vo et al. ( 22 ) demonstrate that state-of-the-art VLMs like Open AI’s o3 16 and Google DeepMind’s Gemini 2.5 Pro 7 exhibit strong prior knowledge biases that severely compromise counting accuracy, as models default to memorized patterns rather than analyzing visual features. This bias extends to general visual understanding. Guo et al. 8 systematically show that VLMs struggle to count beyond 20 objects, with performance deteriorating as scene complexity increases. Aghisi at al. 1 identify that reasoning chains can partially improve counting, with middle transformer layers being most critical for accurate enumeration. Attention mechanisms play a crucial role in VLM visual processing. Kang et al. 12 reveal that VLMs often allocate excessive attention to irrelevant sections of the image and propose a method to redistribute the attention to more relevant areas. An et al. 3 demonstrate that object hallucinations can be mitigated through specialized attention mechanisms that better integrate global and local visual features. Furthermore, synthetic data has proven valuable for controlled VLM evaluation. Several studies have examined the vision understanding of VLMs by evaluating their Visual Question Answering (VQA) capabilities using synthetic imaging data 10 ; 14 ; 15 or by visualizing Attention Guided Class Activation Maps (AG-CAM) on chart-based data 6 . Our work builds upon these foundations by providing a more granular, diagnostic benchmark. We systematically isolate the impact of visual properties (e.g., textures, colors, shapes), allowing us to pinpoint failure modes related to visual complexity rather than just compositional logic. We also implement a set of primitive interventions (e.g., Amplify, Suppress) to better understand the mechanisms of counting failure and to evaluate their potential to improve counting accuracy. 3 Methodology 3.1 Synthetic Evaluation Dataset We create a collection of synthetic datasets, each consisting of images and corresponding prompts, to facilitate a systematic examination of VLMs. Each dataset is designed to evaluate distinct aspects of the input data—both visual and textual. Our generation process begins with a baseline dataset consisting of 512×512 pixel images containing non-overlapping black circular objects on a pure white background. We generate 50 images for this initial configuration. From this baseline, we iterate by varying two primary factors: 1. Object Numerosity: We vary the object counts in buckets of 10, ensuring each bucket is equally represented. This allows us to evaluate how VLM performance degrades as the number of objects successively increases. 2. Visual Properties: We employ a controlled variable methodology. While preserving the object locations from the base dataset, we systematically vary only one feature at a time