Contextual Inference from Single Objects in Vision-Language Models

Martina G. Vilas 1 Timothy Schaumlöffel 1,2* Gemma Roig 1,2 1 Goethe University Frankfurt 2 The Hessian Center for AI · 2026

Read Contextual Inference from Single Objects in Vision-Language Models by Martina G. Vilas 1 Timothy Schaumlöffel 1,2* Gemma Roig 1,2 1 Goethe University Frankfurt 2 The Hessian Center for AI (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26731.txt.

Introduction

Vilas, Schaumlöffel, and Roig ask how much scene context a single object carries inside vision-language models. They show models lone objects on masked backgrounds and test two skills: naming the fine scene type and the broad indoor-outdoor class. Both skills beat chance, and the same object traits that help humans also help the models. Skill at one level does not promise skill at the other, and the coupling varies by model. Objects whose inner form stays steady without a background support the best guesses.

Catalog record

CREATORMartina G. Vilas 1 Timothy Schaumlöffel 1,2* Gemma Roig 1,2 1 Goethe University Frankfurt 2 The Hessian Center for AI
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42521
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26731

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Contextual Inference from Single Objects in Vision-Language Models Martina G. Vilas 1 Timothy Schaumlöffel 1,2* Gemma Roig 1,2 1 Goethe University Frankfurt 2 The Hessian Center for AI Abstract How much scene context a single object carries is a well-studied question in human scene perception, yet how this capacity is organized in vision-language models (VLMs)s remains poorly understood, with direct implications for the robustness of these models. We investigate this question through a systematic behavioral and mechanistic analysis of contextual inference from single objects. Presenting VLMs with single objects on masked backgrounds, we probe their ability to infer both fine-grained scene category and coarse superordinate context (indoor vs. outdoor). We found that single objects support above-chance inference at both levels, with performance modulated by the same object properties that predict human scene categorization. Object identity, scene, and superordinate predictions are partially dissociable: accurate inference at one level neither requires nor guarantees accurate inference at the others, and the degree of coupling differs markedly across models. Mechanistically, object representations that remain stable when background context is removed are more predictive of successful contextual inference. Scene and superordinate schemas are grounded in fundamentally different ways: scene identity is encoded in image tokens throughout the network, while superordinate information emerges only late or not at all. Together, these results reveal that the organization of contextual inference in VLMs is more complex than accuracy alone suggests, with behavioral and mechanistic signatures dissociating in ways that differ substantially across models. 1 Introduction Figure 1: Experimental design. Each image condition (full scene or object-only) is paired with three prompt types: (A) scene category, (B) superordinate category, and (C) object identity. We analyze the intermediate image token representations corresponding to the objects (shown in black). A central question in scene perception research is whether contextual understanding is an emergent property of object representations. Cognitive studies have shown that humans can rapidly infer contextual information from single objects, and this capacity is modulated by object properties that reflect the statistical structure of object-scene relationships in natural images ( 12 ) . Whether the same holds for artificial multimodal systems remains largely unexplored. Understanding how VLMs perform contextual inference from single objects is not only a theoretical question but one with direct practical consequences. It is essential for diagnosing model failures, assessing generalization, and building more robust VLMs. We address this question through a behavioral and mechanistic investigation. We present two VLMs with images of single objects either embedded in their full scene or isolated on a masked background (see Figure 1 ). We probe the behavior and internal representations of these models across two levels of contextual abstraction – scene category and superordinate category (indoor vs. outdoor) – to ask: • Do single objects carry scene-level and superordinate information independently of background context? • Is contextual inference modulated by the same object properties that predict human scene categorization? • Is contextual inference partially independent of object identification? • Are scene-level and superordinate schemas processed through the same internal pathways, or do they arise from distinct mechanisms? • Is contextual information encoded in object-patch representations, and how does the presence or absence of background context modulate these representations? Our results show that single objects activate scene and superordinate schemas, with performance modulated by the same object properties that predict human scene categorization. However, we found important distinctions in how this capacity is organized and grounded across tasks, object properties, and models. Scene and superordinate inference draw on different object properties and are grounded differently in image token representations. Contextual inference is partially independent of object identification, and the two models differ substantially in how tightly object, scene, and superordinate information are integrated. 2 Related Work Contextual effects on object recognition in VLMs. The effect of contextual information on object recognition in VLMs has been previously studied. 8 ask whether VLMs exhibit human-like contextual facilitation: if objects embedded in coherent scenes are recognized better than objects in phase-scrambled scenes, as they are for humans. Similarly, 4 show that contextual incongruity (objects appearing in unexpected scenes) systematically degrades object recognition in VLMs, with models hallucinating contextually expected but absent objects. 6 examine how scene context around an object affects the model’s language generation about that object, and uses attention patterns as a proxy for mechanistic insight. We go in the opposite direction: we remove the scene and ask what the object alone tells the model about its contect, and explain how this inference is grounded mechanistically. Mechanistic Interpretability of VLMs. Previous work has found that image token representations in VLMs become progressively interpretable in the vocabulary space, with object information spatially localized in tokens corresponding to the object’s image region ( 7 ; 9 ) . We adopt the same technique to investigate how contextual information is grounded in object tokens when the scene background is absent. 3 Methods 3.1 Models and Stimuli We evaluate two widely used large vision-language models (VLMs): LLaVA 1.5-13B ( 5 ) and InternVL3.5-14B ( 11 ) (see Appendix A for details). We use a curated subset of the dataset introduced by 2 , comprising N=2{,}392 object-scene pairs drawn from 1{,}004 unique images across eight scene categories (bathroom, bedroom, kitchen, living room, coast, forest, mountain, and skyline). Each image contains multiple objects, but each instance selected for analysis focuses on a single foreground object for which a segmentation mask is available. Full details on the instance selection procedure are provided in Appendix B . For each object instance we record four scene-relevant properties previously investigated in human scene categorization research ( 12 ) : • Frequency : P(\text{object}|\text{scene}) – the proportion of images within the associated scene category that contain the object, irrespective of how many times the object appears within a single image. • Specificity : P(\text{scene}|\text{object}) – the proportion of images containing the object that belong to the associated scene category, capturing how exclusively the object predicts that scene. • Size : The fraction of the image area covered by the object mask. • Object Type : Following 12 , we distinguish between two object types. Anchor objects are large, typically stationary objects that predict the presence and location of smaller objects in their vicinity (e.g., a stove predicting the presence of a pan). Local objects, by contrast, are the smaller objects whose position and identity are predicted by anchors (e.g., the pan). We extract the object type annotations from 10 . Frequency and specificity are computed using the ADE20K dataset ( 14 ) , as its substantially larger size provides more reliable estimates of object–scene co-occurrence statistics. 3.2 Experimental Conditions Each image is presented to each VLM under two viewing conditions: (1) Full scene : the original unmodified image; (2) Object only : only the object is shown, with the rest of the image replaced by a grey background. 3.3 Tasks The VLMs are queried with three successive forced-choice prompts, constructed independently per image. All prompts use an enumerated format in which answer options are listed with numerical indices, presented in randomized order. Model responses were generated greedily, to ensure reproducibility. Full prompt templates are provided in Appendix C . The tasks are: (1) Scene classification: The model is asked to infer the

Read the complete text