Same or Not? Enhancing Visual Perception in Vision-Language Models

Damiano Marsili, Aditya Mehta, Ryan Y. Lin, Georgia Gkixoari · 2025

Read Same or Not? Enhancing Visual Perception in Vision-Language Models by Damiano Marsili, Aditya Mehta, Ryan Y. Lin, Georgia Gkixoari (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23592.txt.

Introduction

Damiano Marsili, Aditya Mehta, Ryan Lin, and Georgia Gkioxari target a blind spot of vision-language models: they recognize broad categories well but miss fine visual details and fall back on biases. Training data shares the blame, since it usually asks only coarse questions, such as whether a picture shows a cat or a dog.

Their answer is TWIN, a data set of more than half a million pairs of similar images, each asking whether the two pictures show the same object. Fine-tuning on TWIN sharpened fine-grained recognition even in domains the training never covered, including art, animals, plants, and landmarks. A new benchmark, FGVQA, with twelve thousand queries, confirmed gains of up to about nineteen percent while general question-answering performance held steady. Scale, their analysis shows, drives the improvement.

Catalog record

CREATORDamiano Marsili, Aditya Mehta, Ryan Y. Lin, Georgia Gkixoari
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Same or Not? Enhancing Visual Perception in Vision-Language Models Damiano Marsili, Aditya Mehta, Ryan Y. Lin, Georgia Gkixoari Abstract Vision–language models (VLMs) excel at broad visual understanding but remain coarse-grained, exhibit visual biases, and miss subtle visual details. Existing training corpora reinforce this limitation by emphasizing general recognition (“Is it a cat or a dog?”) over fine-grained perception. To address this, we introduce a new training corpus and task designed to enhance the perceptual abilities of VLMs. Twin is a large-scale dataset of 561,000 image-pair queries that task models to determine whether two visually similar images depict the same object, encouraging attention to nuanced visual cues. The dataset spans a diverse range of everyday objects across contexts, viewpoints, and appearances. Fine-tuning VLMs on Twin yields notable gains in fine-grained recognition, even on unseen domains such as art, animals, plants, and landmarks. To quantify these gains, we introduce Fgvqa , a benchmark suite of 12,000 queries that repurposes fine-grained recognition and retrieval datasets from multiple domains. While existing VLMs struggle on Fgvqa , when fine-tuned on Twin they improve by up to 19.3 % 19.3\% , without compromising performance on general VQA benchmarks. Finally, our Twin dataset scales favorably with object annotations, and our analysis shows that scale is key to performance. We envision Twin as a drop-in addition to open-source VLM training corpora, advancing perceptual precision of future models. Project webpage: https://glab-caltech.github.io/twin/ (a) The Twin Dataset (b) The Fgvqa Benchmark Suite Figure 1 : (a) Fine-grained visual understanding requires recognizing subtle differences between similar images. We introduce Twin , a large-scale dataset of 561K image-pair queries asking if two images depict the same object, and Fgvqa , a benchmark suite evaluating fine-grained VQA capabilities. (b) Models trained on Twin show improved fine-grained understanding on Fgvqa , even in unseen domains. 1 Introduction Fine-grained visual understanding – recognizing subtle details within images – is a hallmark of human vision and a desirable goal for machine perception. Modern vision-language models (VLMs) have shown tremendous progress in broad visual reasoning, however their perception remains coarse-grained: they exhibit systematic biases 36 ; 71 ; 86 ; 87 and overlook subtle details 18 ; 66 . Fig. 1 (a) showcases these challenges. While the vacuum cleaners share the same color and brand name, they are distinct instances, distinguishable by the geometry of their dustbins, handle, and base. Qwen2.5-VL 5 , a strong open-source VLM, incorrectly identifies them as identical and reveals flawed reasoning about their visual attributes. We attribute the limitations of current VLMs in fine-grained perception partly to their training data. Most large-scale image–text corpora emphasize general visual reasoning – such as spatial relations 29 ; 50 ; 26 ; 80 , common knowledge 105 ; 49 , grounding 34 ; 63 ; 82 ; 30 ; 107 , or mathematical reasoning 3 ; 45 ; 109 ; 90 – over detailed visual discrimination. While these datasets enable broad understanding, they provide little incentive to attend to subtle, instance-level differences. This imbalance is stark in leading open-data VLMs 9 ; 48 , where the majority of their training data does not reward visual distinctions beyond category-level understanding. We ask: what if VLMs were trained on data that rewards fine-grained visual understanding? To explore this question, we introduce Twin , a large-scale VQA dataset of 561{,}000 instance-centric queries. We name our dataset Twin as it features TWo-image INstance comparisons: each query asks whether two images depict the same physical object instance, requiring attention to subtle visual cues beyond category-level semantics. We source images of household objects across diverse backgrounds, viewpoints, contexts, and lighting conditions, verified by human annotators. We focus on household objects as their images are readily available and they are central to common visual tasks such as robotics and embodied interaction. Fig. 1 (a) shows a sample from Twin and illustrates the challenge: distinguishing the vacuum cleaners requires noting differences in form factor and shape, rather than shared brand or color. To evaluate progress in fine-grained visual perception in VLMs, we introduce Fgvqa , a benchmark suite for fine-grained VQA. While fine-grained recognition tasks have long existed in vision research, they are typically formulated as 1-of-N classification or retrieval problems, making them ill-suited for VLMs. Fgvqa adapts and reformulates existing recognition and retrieval datasets for VLM evaluation, comprising 12{,}000 queries from diverse domains: Met 104 with artworks such as paintings and sculptures, Inquire 85 featuring animals and plants from iNaturalist 84 , Cub 88 with bird species, Ilias 32 showcasing retail products, and Landmarks 94 containing landmarks such as monuments and architectural structures. VLMs post-trained on Twin show substantial gains on Fgvqa , as illustrated in Fig. 1 (b) which compares the base and post-trained Qwen2.5-VL. Notably, the gains are substantial even on domains that span beyond Twin , e.g . in Inquire and Cub . Through qualitative analyses of model explanations and probing experiments on the vision encoder, we find that VLMs trained on Twin attend more effectively to nuanced, distinguishing visual cues. Finally, our scaling analysis shows that scale is crucial for performance, establishing Twin , which scales in size favorably with the number of object annotations, as a promising path for advancing perceptual precision in VLMs. 2 Related Work Limitations of VLMs. VLMs trained on large text-image corpora have shown remarkable capabilities in image understanding. Here, proprietary systems 60 ; 1 ; 12 ; 4 lead in performance, while open-source alternatives 41 ; 96 ; 15 ; 14 ; 53 ; 48 ; 79 are narrowing the gap. Despite these advances, VLMs exhibit systematic biases 86 ; 36 ; 87 ; 71 , struggle with fine-grained visual perception 18 ; 66 , and often hallucinate details 39 ; 81 ; 95 ; 102 . To address these challenges, we introduce a large-scale VQA dataset to enhance fine-grained visual understanding in VLMs along with a benchmark suite to evaluate this capability. VLM Training Data. While most proprietary VLMs rely on closed-source training data 60 ; 1 ; 4 ; 12 ; 77 , recent efforts have prioritized open-data alternatives. Molmo 14 introduced an open-data model competitive with proprietary systems, while SmolVLM 48 and PerceptionLM 9 scale this paradigm using diverse datasets spanning captioning 42 ; 2 ; 103 ; 59 ; 100 , grounding 34 ; 63 ; 82 ; 30 ; 107 , spatial reasoning 18 ; 29 ; 43 , math 46 ; 11 ; 3 , and chart/diagram understanding 52 ; 58 ; 31 ; 76 ; 92 . Most relevant to us are SpotTheDiff 28 and Birds-to-Words 17 , which pair images with text describing differences. In contrast, Twin is more comprehensive and larger in size: Twin features both matching and non-matching pairs, enabling reasoning about sameness and difference, distinctions in Twin are finer and pertain to subtle cues, Twin is far larger ( 561 K pairs vs. 13 K of SpotTheDiff and 3.3 K of Birds-to-Words), and it spans diverse object types beyond a single domain such as birds. A detailed comparison is in the Appendix. Fine-grained Recognition categorizes images into specific subgroups 93 , with applications in classifying plants 84 ; 55 ; 85 , birds 89 ; 83 , vehicles 47 ; 33 and

Read the complete text