Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs

Aditya Kanade, Tanuja Ganu · 2025

Read Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs by Aditya Kanade, Tanuja Ganu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02022.txt.

Introduction

Aditya Kanade and Tanuja Ganu built a benchmark in two thousand twenty-five to test how well multimodal language models see. Such models can give a right answer while still misreading key parts of an image. That hidden flaw can make their reasoning look stronger than it is.

The benchmark, called Do You See Me, holds one thousand seven hundred fifty-eight images and two thousand six hundred twelve questions. Its seven subtasks draw on human psychology and span two- and three-dimensional scenes, with set levels of difficulty. Humans scored near ninety-six percent, while top models averaged below fifty percent. The gap grew fast as tasks got harder. The authors trace many errors to weak attention and unstable fine detail.

Catalog record

CREATORAditya Kanade, Tanuja Ganu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE87 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-02022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs Aditya Kanade, Tanuja Ganu Abstract Multimodal Large Language Models (MLLMs) show reasoning promise, yet their visual perception is a critical bottleneck. Strikingly, MLLMs can produce correct answers even while misinterpreting crucial visual elements, masking these underlying failures. Our preliminary study on a joint perception-reasoning dataset revealed that for one leading MLLM, 29% of its correct answers to reasoning questions still exhibited visual perception errors. To systematically address this, we introduce Do You See Me , a scalable benchmark with 1,758 images and 2,612 questions. It spans seven human-psychology inspired subtasks in 2D and 3D, featuring controllable complexity to rigorously evaluate MLLM visual skills. Our findings on 3 leading closed-source and 5 major open-source models reveal a stark deficit: humans achieve 96.49% accuracy, while top MLLMs average below 50%. This performance gap widens rapidly with increased task complexity (e.g., from 12% to 45% in the visual form constancy subtask). Further analysis into the root causes suggests that failures stem from challenges like misallocated visual attention and the instability of internal representations for fine-grained details, especially at or below encoder patch resolution. This underscores an urgent need for MLLMs with truly robust visual perception. The benchmark dataset, source code and evaluation scripts are available at 1 1 1 https://github.com/microsoft/Do-You-See-Me 1 Introduction Visual perception—the ability to interpret and understand visual stimuli—forms the foundation for higher-order cognitive tasks such as visual reasoning. Recent advances in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities across diverse domains [ 25 , 47 , 31 , 43 ] . As a result, visual question-answering (VQA) tasks have become critical benchmarks for assessing how well MLLMs integrate visual perception with higher-level reasoning [ 44 ] . However, multiple studies have pointed out that current MLLMs often struggle with fine-grained disambiguation of visual information required for accurate reasoning [ 47 ] . Figure 1 illustrates examples where Claude Sonnet-3.5 [ 4 ] and GPT-4o [ 26 ] exhibit visual perception errors while solving a logical pattern-completion task. Notably, Claude Sonnet-3.5 arrives at the correct final answer despite misinterpreting critical visual elements—highlighting that correct reasoning outcomes may mask underlying perception failures . This observation underscores the need for benchmarks that independently evaluate visual perception skills in MLLMs. Datasets like MMVP [ 37 ] , CV-Bench [ 35 ] , and MVP-Bench [ 17 ] offer insights into visual capabilities. However, their reliance on detailed human annotations limits scalability. Moreover, explicit evaluation of visual perception remains a minor component within these datasets. Thus, there remains a pressing need for a scalable, perception-focused evaluation framework. To guide our investigation into MLLM visual perception, this study aims to answer the following research questions: • RQ1: Does an MLLM’s ability to correctly answer a reasoning question based on a visual diagram necessarily indicate accurate visual perception of that diagram’s key elements? • RQ2: How proficient are current MLLMs on core visual perception tasks of varying complexity, particularly when compared to human performance? • RQ3: What are the underlying structural limitations or failure modes in current MLLMs that contribute to their deficits in visual perception? Investigating Reasoning vs. Perception Deficiencies: To motivate the development of a perception-focused benchmark, we first design a joint perception-reasoning dataset. This preliminary dataset aims to expose deficiencies in the visual perception abilities of MLLMs. We curate 150 unique images drawn from logic-based Intelligence Quotient (IQ) tests, which emphasize pattern recognition and "pure" reasoning with minimal domain-specific knowledge. Our dataset is constructed by merging samples from MathVista [ 25 ] and LogicVista [ 43 ] , covering both perception- and reasoning-oriented questions for each image. Our experiments on the joint perception-reasoning dataset reveal that relying solely on final-answer accuracy can obscure critical perceptual shortcomings. For instance, Claude Sonnet-3.5 achieves a top reasoning accuracy of 41%, yet 29% of its correctly answered samples still exhibit visual perception errors. Conversely, among incorrectly answered samples, both reasoning and perception errors are observed in equal proportion. These findings emphasize the need to independently assess perception independent to reasoning in MLLMs. Figure 1 : Visual Misinterpretations in Popular Multimodal LLMs A Multidimensional Visual Perception Benchmark: Building upon frameworks in human psychology [ 7 ] , where visual perception is categorized into five core skills—visual discrimination, figure-ground perception, spatial relations, closure, and sequential memory—we introduce the Do You See Me benchmark. This programmatically generated suite comprises 1,758 unique images and 2,612 evaluation questions, systematically mapped to seven subtasks aligned with fundamental visual perception abilities. Notably, most of these subtasks are presented in both 2D (SVG-based) and photorealistic 3D rendered settings, allowing for evaluation across varied visual complexities. Control parameters for each subtask ( Table 2 ) allow fine-grained modulation of difficulty levels. Because Do You See Me is fully synthetic, it enables scalable generation of diverse Image-QA pairs and facilitates detailed analysis of MLLMs’ perceptual strengths and weaknesses. Visual Perception in MLLMs: Our comprehensive evaluations using the Do You See Me benchmark starkly reveal the current limitations of MLLM visual perception. Across the seven designed subtasks, human participants demonstrated robust performance, achieving an average accuracy of 96.49%. In stark contrast, even the leading MLLMs lagged significantly, with the top-performing model averaging only 50.05% accuracy, and no single model consistently excelling across all perceptual challenges. This performance chasm widens alarmingly with increased task complexity. These findings point to substantial deficiencies in the foundational visual perception skills of contemporary MLLMs. Uncovering Structural Limitations: Delving deeper, we uncover several underlying factors contributing to these perceptual failures. We identify critical issues such as MLLMs’ misallocated visual attention towards query-irrelevant regions and an instability in their internal representations of fine-grained visual details, especially for stimuli near or below their visual encoder’s patch resolution. Furthermore, we find that common strategies like supervised finetuning on distributionally similar data yield only modest gains in addressing these core perceptual deficits. These insights collectively suggest that the observed shortcomings are not superficial but stem from fundamental limitations in how MLLMs currently process, interpret, and ground visual information. This study therefore not only introduces a novel, scalable, and psychologically-grounded benchmark but also provides a detailed characterization of these critical perceptual weaknesses, underscoring the urgent need for innovative approaches to cultivate more genuinely perceptive MLLMs. In summary, our key contributions are: 1. A programmatically generated dynamically scalable benchmark— Do You See Me —designed to systematically measure core visual perception skills inspired by human psychology. 2. A curated joint dataset pairing logic-based reasoning questions with closely related visual perception tasks. 3. An extensive error analysis of multiple state-of-the-art MLLMs, including manual annotation of reasoning chains to identify perception failures. 4. A comprehensive investigation of MLLM visual perception performance on Do You See Me benchmark dataset, and identification of reasons behind failure. 2 Related Work Multimodal Large Language Models (MLLMs) have evolved from joint image-language pretraining approaches, exemplified by CLIP [ 29 ] , BLIP [ 18 , 19 ] , and SigLIP [ 46 ] . With the rapid progress of large language models (LLMs), the ’vision as a token’ paradigm has emerged, fostering numerous closed and open-source MLLMs that continue to advance through large-scale data and instruction finetuning [ 27 , 42 ,

Read the complete text