The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs
Tejas Anvekar, Fenil Bardoliya, Pavan K. Turaga, Chitta Baral, Vivek Gupta · 2025
Read The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs by Tejas Anvekar, Fenil Bardoliya, Pavan K. Turaga, Chitta Baral, Vivek Gupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15949.txt.
Introduction
Tejas Anvekar and co authors present the Perceptual Observatory, a framework for characterizing perception in multimodal language models, in a 2025 paper. Many model families scale the language part while reusing nearly identical vision encoders. Leaderboard accuracy cannot tell whether gains reflect true visual grounding or borrowed text knowledge.
The observatory tests models on simple vision tasks such as face matching and reading text inside images. It also tests local to global skills: image matching, a grid pointing game, and attribute localization. Ground truth face and word data are perturbed with pixel changes and diffusion based illusions. The framework probes robustness and attribution fidelity rather than end task scores alone.
Catalog record
| CREATOR | Tejas Anvekar, Fenil Bardoliya, Pavan K. Turaga, Chitta Baral, Vivek Gupta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15949
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs Tejas Anvekar, Fenil Bardoliya, Pavan K. Turaga, Chitta Baral, Vivek Gupta Abstract Recent advances in multimodal large language models (MLLMs) have yielded increasingly powerful models, yet their perceptual capacities remain poorly characterized. In practice, most model families scale language component while reusing nearly identical vision encoders (e.g., Qwen2.5-VL 3B/7B/72B), which raises pivotal concerns about whether progress reflects genuine visual grounding or reliance on internet-scale textual world knowledge. Existing evaluation methods emphasize end-task accuracy, overlooking robustness, attribution fidelity, and reasoning under controlled perturbations. We present The Perceptual Observatory , a framework that characterizes MLLMs across verticals like: (i) simple vision tasks, such as face matching and text-in-vision comprehension capabilities; (ii) local-to-global understanding, encompassing image matching, grid pointing game, and attribute localization, which tests general visual grounding. Each vertical is instantiated with ground-truth datasets of faces and words, systematically perturbed through pixel-based augmentations and diffusion-based stylized illusions. The Perceptual Observatory moves beyond leaderboard accuracy to yield insights into how MLLMs preserve perceptual grounding and relational structure under perturbations, providing a principled foundation for analyzing strengths and weaknesses of current and future models. 1 Introduction Figure 1 : Overview of The Perceptual Observatory and how it solicits understanding of opaque MLLMs perceptual understanding by measuring properties motivated by human visual perception and robustness against multiple axes. We illustrate the framework for properties revealing true perceptual understanding of MLLMs Multimodal Large Language Models (MLLMs) are ubiquitous for tasks such as captioning, VQA, OCR-centric reasoning, document understanding, accessibility, robotics, and multi-image dialogue [ 29 , 3 , 31 , 1 , 9 , 7 ] . Public leaderboards (e.g., MMBench, MMMU/Pro; TextVQA; VizWiz; SEEDBench; POPE; MATHVista) mostly report end-task accuracy [ 32 , 53 , 41 , 16 , 28 , 27 , 26 , 30 , 33 ] . However, outstanding benchmark performance does not guarantee robust perception – defined as the fundamental ability to faithfully understand and interpret visual details, maintain object identity, and spatially ground independent of linguistic reasoning. Without this, models can exploit textual priors, miss identity under perturbations, or fail to localize evidence. Modern MLLMs scale the language side while leaving vision encoders frozen or lightly adapted via compact bridges (Qwen2.5-VL-family, Gemma3-family, Q-Former, Perceiver resampler, MLP/linear projectors) [ 43 , 42 , 3 , 31 ] . This raises the question of whether the gains are due to better visual or better textual capabilities? Decades of vision research warns that models can rely on shortcuts; language priors in VQA or texture bias in CNNs masks poor perceptual grounding [ 15 , 2 , 12 ] . Furthermore, while web-scale pretraining increasingly obscures the boundary between In-distribution (ID) and out-of-distribution (OOD) data, foundational robustness studies demonstrate that accuracy can precipitously decline under even modest corruptions or distribution shifts [ 21 , 22 , 5 ] Based on human-cognitive behaviour, where perception remains robust across stylistic variations and environmental noise [ 48 , 49 , 13 , 25 ] , we probe the depth of machine seeing against these biological standards. We then ask: (Q1) Do MLLMs preserve identity under content-preserving ID corruptions and under OOD stylized images? (Q2) Are predictions positional-invariant when the same content moves in a grid? (Q3) Do models ground attributes where they belong, and does giving hints improve transfer? (Q4) Does scaling primarily on the language side yield monotonic perceptual gains when the vision encoder is fixed? (Q5) Does enabling <think> mode materially facilitate perception, or just the narrative? (Q6) Are there fairness gaps across subpopulations (e.g., gender, race, lighting, texture) under shifts? For addressing the aforementioned research questions, we introduce The Perceptual Observatory , a holistic evaluation suite that measures how MLLMs see. We probe with (i) ID augmentations and (ii) OOD stylized illusions [ 18 ] images produced by diffusion with spatial control (Stable/Latent Diffusion + ControlNet) that alter appearance while preserving layout, letting us disentangle perception from priors [ 38 , 54 ] . Tasks target complementary skills: identity matching (robustness to perturbations vs. distractors), grid pointing game (spatial invariance), and attribute localization for semi and fully guided settings [ 10 ] towards common-sense reasoning [ 11 ] assessment. We summarize our contributions as follows: • We propose The Perceptual Observatory : A principled framework that evaluates perceptual robustness and vision-language grounding beyond tradition benchmark performance, highlighting whether failures stem from visual or textual capabilties. • We consolidate simple, interpretable properties of MLLMs like identity robustness, spatial invariance, attribution fidelity, fairness gap, scale consistency, and effects of <think> mode to reveal how answers are grounded. • To enable further research in this area, we also provide a scalable pipeline to generate ID corruptions and OOD stylized illusions (diffusion+ControlNet) that preserve spatial layout while confounding appearance. • Finally, we provide a comprehensive analysis of three leading open-source MLLM families. We demonstrate that scaling the language model without proportional adaptation of the vision encoder results in systematic robustness gaps under distribution shifts, thereby pinpointing the methodological bottlenecks that future research must address. 2 Related Works With the recent wave of MLLM families such as Qwen2.5-VL [ 43 ] , Gemma3 [ 42 ] , InternVL3.5 [ 51 ] , etc., has dramatically pushed the boundaries of visual perception. The large-scale models have frozen or lightly adapted vision backbones such as ViT [ 8 ] , SigLIP 2 [ 46 ] , CLIP [ 36 ] . The early evaluation of these models has emphasized end-task accuracy. Benchmarks such as MMBench [ 32 ] and MMMU [ 52 ] extend text-centric evaluation to vision language understanding, offering huge collections of diverse QAs (c.f. MMLU [ 20 ] ). Yet, these efforts lack perceptual understanding with language priors, leading to the question of whether the high scores on the benchmarks arise from the visual grounding or from the textual reasoning. The computer vision community has long highlighted the fragility of models under distribution shifts [ 19 , 21 ] . Analogous concerns have emerged for MLLMs. Experiments in abstract shape recognition show that VLMs often rely on texture or contextual clues rather than true shape understanding [ 18 ] . Similarly, [ 40 , 39 , 55 , 17 , 56 ] construct optical illusions and misleading visual scenarios. These works show that MLLMs are easily misled, as they capture end-task accuracy aided by prompting techniques to improve understanding, yet do not close the gap to human performance without explainability. Another flaw is that the models may have already been trained on certain popular illusions, such as Salvador Dali’s painting [ 35 ] . QAs such as CLEVR [ 23 ] and Winoground [ 44 ] reveal that models fail to reason on spatial relations and subtle changes [ 47 ] . Beyond QA, VLMs may produce correct answers while attending to irrelevant regions, highlighting poor vision-language disentanglement. Thus, robust multimodal understanding requires attribution localization. Recent MLLMs predict bounding boxes for attributes, enabling explicit evaluation, but localization under distribution shifts for perturbations and illusion remains scarce. While these prior benchmarks demonstrate critical weaknesses – language-prior exploitation, fragility to corruption, distribution shifts, and poor grounding – they traditionally examine one dimension at a time . The Perceptual Observatory fills this