VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?
Minkyu Kim, Sangheon Lee, Dongmin Park · 2026
Read VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning? by Minkyu Kim, Sangheon Lee, Dongmin Park (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07888.txt.
Introduction
Kim and colleagues introduce VLM-SubtleBench, a test of whether vision-language models can spot small differences between near-identical images. It spans ten difference types, from attribute and state to viewpoint and action, and covers industrial, aerial and medical images beyond natural photos alone. Extensive tests of open and closed models reveal systematic gaps from human performance across types and domains, and pinpoint where model reasoning breaks down most sharply.
Catalog record
| CREATOR | Minkyu Kim, Sangheon Lee, Dongmin Park |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67656 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.07888
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning? Minkyu Kim, Sangheon Lee, Dongmin Park Abstract The ability to distinguish subtle differences between visually similar images is essential for diverse domains such as industrial anomaly detection, medical imaging, and aerial surveillance. While comparative reasoning benchmarks for vision-language models (VLMs) have recently emerged, they primarily focus on images with large, salient differences and fail to capture the nuanced reasoning required for real-world applications. In this work, we introduce VLM-SubtleBench 1 1 1 Our dataset and code are available at https://huggingface.co/datasets/KRAFTON/VLM-SubtleBench and https://github.com/krafton-ai/VLM-SubtleBench . , a benchmark designed to evaluate VLMs on subtle comparative reasoning . Our benchmark covers ten difference types—Attribute, State, Emotion, Temporal, Spatial, Existence, Quantity, Quality, Viewpoint, and Action—and curate paired question–image sets reflecting these fine-grained variations. Unlike prior benchmarks restricted to natural image datasets, our benchmark spans diverse domains, including industrial, aerial, and medical imagery. Through extensive evaluation of both proprietary and open-source VLMs, we reveal systematic gaps between model and human performance across difference types and domains, and provide controlled analyses highlighting where VLMs’ reasoning sharply deteriorates. Together, our benchmark and findings establish a foundation for advancing VLMs toward human-level comparative reasoning. * * footnotetext: Equal contribution. $\dagger$ $\dagger$ footnotetext: Work done during an internship at KRAFTON. 1 Introduction Figure 1: Comparison of VLM-SubtleBench and MLLM-CompBench with GPT-4o. The ability to discern subtle visual differences , i.e., minor discrepancies between otherwise similar objects, scenes, or situations, is central to human cognition. It enables us to perform a wide range of fine-grained comparative tasks, such as recognizing micro-expression changes in daily life ( 29 ) , detecting anomalies in manufacturing ( 7 ) , assessing subtle variations in satellite imagery ( 4 ) , and distinguishing disease stages in medical imaging ( 32 ) . Beyond everyday life, the capacity to detect and reason about such subtle differences has expanded human abilities toward higher forms of intelligence, underpinning advances from scientific discovery to complex social organization. Recently, vision-language models (VLMs) have shown remarkable progress toward artificial general intelligence (AGI), demonstrating promising results in various tasks, such as visual question answering (VQA) and scene description ( 50 ) . Yet, most progress has primarily centered on single visual inputs, e.g., an image or a video, while comparative tasks that require comparison over multiple inputs, e.g., two images, have received relatively little attention. However, narrowing this gap is increasingly important, as recent applications often deploy VLMs as agents to perform complex tasks involving comparative reasoning across multiple observations, e.g., self-reflection over previously observed scenes ( 13 ) . To truly serve as human-level surrogates, advancing VLMs’ capacity for advanced comparative reasoning is becoming indispensable. A few benchmarks have been presented to evaluate the comparative reasoning capabilities of VLMs. However, as shown in Figure 1 , prior benchmarks focus on relatively simple comparisons between fairly dissimilar scenes, e.g., identifying differences in salient features (states) of distinct objects (two lemons), as indicated by the low average subtlety scores of MLLM-CompBench measured by embedding similarity with DINOv3 ( 39 ) . As a result, they are easily solved by state-of-the-art VLMs, such as GPT-4o. In addition, most prior benchmarks are composed of natural images, and thus fail to assess performance in specialized domains such as industry or medical (See Table 1 ). Therefore, this calls for a new benchmark that can evaluate human-level subtle comparative reasoning across diverse domains and high-difficulty tasks. Figure 2: Example tasks from the VLM-SubtleBench, covering ten difference categories (Attribute, State, Emotion, Temporal, Spatial, Existence, Quality, Quantity, Viewpoint, Action) and six domains (natural, game, medical, industry, aerial, synthetic). For each example, the correct answer is highlighted in bold green . Model responses from GPT-5-main, Claude-sonnet-4, and Gemini-2.5-pro are shown beneath each question in order. Some VQA instances are simplified due to space constraints; full versions and additional examples are provided in the appendix. To this end, we present VLM-SubtleBench, a benchmark designed to evaluate human-level comparative reasoning capabilities of VLMs. As illustrated in Figure 2 , VLM-SubtleBench comprises 13K triplets of image pairs, questions, and answers, covering 10 representative difference types, i.e., Attribute , State , Emotion , Temporal , Spatial , Existence , Quality , Quantity , Viewpoint , and Action , collected from diverse image domains, i.e., Natural , Game , Industry , Aerial , Synthetic , and Medical . Even advanced proprietary VLMs such as GPT-5 and Gemini-2.5-pro exhibit significant performance gaps compared to humans, especially on difference types that demand spatial, temporal, and viewpoint reasoning. We conduct systematic studies on the performance of open-source and proprietary VLMs, the effectiveness of test-time prompting strategies on comparative tasks, and controlled experiments with synthetic image pairs, which reveal the extent to which VLMs align with human performance and where their reasoning sharply deteriorates. Our findings show that (1) proprietary VLMs still leave large gaps from human performance, particularly on spatial, temporal, and viewpoint reasoning where even the best model trails humans by over 30 percentage points; (2) simple prompting strategies, such as chain-of-thought prompting, grid layouts, and overlapping images, yield only limited improvements; and (3) VLMs are highly sensitive to difficulty factors such as object size and count. Together, VLM-SubtleBench and our experimental studies provide critical insights toward narrowing the gap between VLMs and humans in subtle comparative reasoning. 2 Related Work Multi-Image Benchmarks for VLMs. While most VLM benchmarks focus on single-image understanding ( 34 ; 49 ; 48 ) , a growing body of work evaluates multi-image reasoning. Early efforts such as NLVR2 ( 42 ) and Winoground ( 44 ) tested whether models can reason over image pairs through sentence verification and compositional matching, respectively. More recently, BLINK ( 11 ) evaluates core visual perception tasks such as relative depth and visual correspondence, exposing large human–model gaps on low-level visual skills. REMI ( 24 ) focuses on high-level relational reasoning, such as analogy and sequential pattern completion, across multiple images. MuirBench ( 45 ) provides a comprehensive suite of 12 multi-image task types, ranging from visual retrieval to scene understanding. While these benchmarks collectively advance multi-image evaluation, none are specifically designed to evaluate comparative reasoning between image pairs, identifying what has changed between two similar images. MLLM-CompBench ( 25 ) is the closest to our work, as it directly evaluates comparative judgments across eight difference types. Yet, its image pairs typically depict different subjects or settings with salient differences, as evidenced by their low average similarity scores (Figure 1 ). In contrast, VLM-SubtleBench focuses on subtle differences within nearly identical contexts, requiring genuine cross-image comparison (Table 1 ). Table 1: Summary of Comparative Reasoning Benchmarks. ‘Is Subtle?’ indicates whether the benchmark contains image pairs whose DINOv3 similarity averages at least 0.8, meaning the differences are subtle. VLM-SubtleBench is the only benchmark that focuses on subtle comparison, spans diverse domains, and includes both multiple-choice questions and captioning tasks. Benchmarks Is Subtle? # Domains # Diff. Types MCQ Captioning Birds-to-Words ( 10 ) ✗ 1 1 ✗ ✓ Spot-the-Diff ( 20 ) ✓ 1 1 ✗ ✓ MLLM-CompBench ( 25 ) ✗ 1 8 ✓ ✗ VLM-SubtleBench (Ours) ✓ 6 10 ✓ ✓ Difference Understanding across Domains. Detecting and describing visual differences has been studied across