COOCO - Common Objects Out-of-Context - Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication

Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt · 2025

Read COOCO - Common Objects Out-of-Context - Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication by Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22274.txt.

Introduction

Filippo Merlo, Ece Takmaz, Wenkai Chen, and Albert Gatt ask how much vision-language models lean on scene context when they refer to objects. They build a data set of common objects placed in scenes where they fit well or badly. That setup lets them test how context and noise shape model replies.

Models used context in a flexible way, depending on how well an object fit the scene and how much noise was present. Good naming went with stronger attention to the target in middle layers of the model. Attention also rose for both very low and very high fit. The results show a balance between local detail and whole-scene cues.

Catalog record

CREATORFilippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-22274

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

COOCO - Common Objects Out-of-Context - Semantic Violation in Scenes: Investigating Multimodal Context in Referential Communication Filippo Merlo, Ece Takmaz, Wenkai Chen, Albert Gatt Abstract Natural scenes provide us with rich contexts for object recognition and reference. In particular, knowing what type of scene one is looking at generates expectations about which objects will occur, and what their spatial configuration should be. Do Vision-Language Models (VLMs) learn to rely on scene contexts in a similar way, when generating references to objects? To address this question, we introduce the Common Objects Out-of-Context (COOCO) dataset and test to what extent VLMs rely on scene context to refer to objects under different degrees of scene-object congruency, and different perturbations. Our findings show that models leverage scene context adaptively, depending on both the semantic relatedness between object and scene and the level of noise. In particular, models rely more on context under high target-scene congruence or when objects are degraded. Attention analysis reveals that successful object categorisation involves increased focus on the target in mid-level layers, especially under moderate noise, suggesting that VLMs dynamically balance local and contextual information for reference generation. We make our dataset, code and models available at https://github.com/cs-nlp-uu/scenereg . 1 Introduction Objects rarely appear in isolation. Over time, humans build expectations of seeing certain objects in certain contexts. For instance, we expect to see a laptop on a desk in an office, but not a large piece of ham (Figure 1 ). Studies of human perception in visual scenes Võ 2021 ; Torralba et al. 2006 ; Coco et al. 2016 suggest that scenes exhibit both semantic and syntactic regularities Biederman et al. 1982 ; Võ 2021 . Violations of either or both of these result in processing difficulties Ganis and Kutas 2003 ; Öhlschläger and Võ 2016 ; Võ 2021 . Original Modified Context Noise Target Noise Figure 1 : An original image of an office scene from the COCO dataset and modified version from our dataset, in which the laptop is replaced with an object that has low relevance to the scene. Bottom: versions of the modified image with Gaussian noise mask (level = 0.5) on the context and on the target. In this work, we are interested in how Vision-Language Models (VLMs) behave in the presence of semantic violations in visual scenes when referring to objects. Such an analysis offers insights into the extent to which models learn to leverage contextual visual information. We focus on object naming , that is, the task of determining the category of an object, a fundamental part of Referring Expression Generation ( Krahmer and Van Deemter 2012 , REG;) . The literature on scene perception suggests that scene semantics mediates human viewers’ expectations about the objects in a scene, and viewers deploy their attentional resources accordingly. Scene semantics also influence how humans describe objects in scenes Hwang et al. 2011 ; de Groot et al. 2017 ; de Groot et al. 2015 . Do VLMs similarly utilise scene-level visual information when generating descriptions of objects? To study this, we compare the naming of objects in semantically (in)congruent scenes. If scene-level information guides model choices, then there should be systematic differences between these two conditions in the way models refer to target referents. Previous work has shown the facilitating effects of context in REG, even in the case of perturbations made on images ( Junker and Zarrieß 2024 ) . However, this work does not control for the impact of semantic (in)congruity, and relies on small-scale, purposely-trained models or on artificial images ( Junker and Zarrieß 2025 ) . We develop a new dataset, Common Objects Out-of-Context ( COOCO ), which manipulates natural images to introduce objects of varying degrees of consistency with the scene (Figure 1 ). Similarly to Junker and Zarrieß 2024 , we further manipulate the visibility of the target referent by introducing different degrees of noise. We summarise the contributions of our study as follows. 1. We present COOCO, a novel dataset to evaluate multimodal models’ capabilities for integrating object-level and scene-level visual information, under both congruent and incongruent conditions. 2. We investigate the impact of scene context on object identification in state-of-the-art VLMs. We show that under conditions where target objects violate scene semantics, scene context acts as a distractor, while it has a clear facilitating effect when the target is congruent with the scene. This also makes models resilient to noise when it is applied to the target region. 3. We investigate how attention is distributed by VLMs across scene components under different conditions of visual semantic congruency, when generating referring expressions. 2 Related Work Context in Referring Expression Generation. Referring Expression Generation (REG) is the task of generating identifying descriptions for a target object. Early REG approaches relying on handcrafted algorithms Krahmer and Van Deemter 2012 incorporated contextual influences by considering the properties that distinguish a target referent from its distractors, with some research also incorporating constraints to guide coherent categorisation for references to plurals, and the use of landmarks for spatial expressions Schüz et al. 2023 . More recent neural models for visual REG learn mappings from raw perceptual inputs to linguistic expressions. Here too there has been limited attention on the ability of models to leverage global scene information Schüz et al. 2023 . This gap has motivated recent investigations into how scene information may enhance robustness in challenging settings Schüz and Zarrieß 2023 ; Junker and Zarrieß 2025 . In particular, Junker and Zarrieß 2024 show that models rely on scene context to generate references even when the target object is fully occluded, indicating that scene context serves not only as a disambiguating constraint but also as a supportive resource for robust reference generation. Context in Human Object Recognition. Human object recognition is strongly influenced by contextual expectations Bar 2004 ; Oliva and Torralba 2007 ; Võ 2021 ; Peelen et al. 2023 , including both global scene properties (e.g., gist, layout) and local cues (e.g., anchor objects, co-occurring items) Lauer et al. 2018 ; Lauer et al. 2021 ; Boettcher et al. 2018 . Evidence that visual search is semantically constrained comes from viewing tasks, where viewers fixate more on semantically informative (but not necessarily visually salient) regions, and semantic similarity is a strong predictor of gaze trajectories. Both highly expected and highly incongruent objects attract more attention than moderately fitting ones Damiano et al. 2024 . Datasets. Widely used vision and language datasets for referring expression and image caption generation, such as COCO Lin et al. 2015 , RefCOCO Kazemzadeh et al. 2014 ; Yu et al. 2016a and VisualGenome Krishna et al. 2016 , lack annotations for object-scene relatedness and do not enable research on semantic violations in scenes. A number of scene violation datasets have been created within the vision community. Prominent examples include ObjAct Shir et al. 2021 , Event Task Dresang et al. 2019 , Out-of-Context Bomatter et al. 2021 , Cut-and-paste dataset Yun et al. 2021 , and SCEGRAM Öhlschläger and Võ 2016 . However, these datasets tend to be relatively small-scale and do not incorporate degrees of semantic violation or occlusion. In this work, we present a new, large-scale dataset for semantic violations in

Read the complete text