MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models Anh Thai, Stefan Stojanov, Zixuan Huang, Bikram Boote, James M. Rehg Abstract This paper introduces MEBench, a novel benchmark for evaluating mutual exclusivity (ME) bias, a cognitive phenomenon observed in children during word learning. Unlike traditional ME tasks, MEBench further incorporates spatial reasoning to create more challenging and realistic evaluation settings. We assess the performance of state-of-the-art vision-language models (VLMs) on this benchmark using novel evaluation metrics that capture key aspects of ME-based reasoning. To facilitate controlled experimentation, we also present a flexible and scalable data generation pipeline that supports the construction of diverse annotated scenes. Figure 1 : Mutual Exclusivity Bias Evaluation Settings. (a) Traditional ME bias evaluation in developmental psychology and early computational studies [ 6 ] , (b) MEBench setup for classic ME bias testing, and (c) MEBench setup for evaluating ME bias in conjunction with spatial reasoning. 1 Introduction Toddlers are remarkable learners, acquiring 10–20 new words per week during a developmental phase known as the vocabulary spurt [ 16 ] . This rapid word learning is supported by powerful inductive biases that help children resolve the ambiguity of mapping spoken words to real-world referents, referred to as object-label mapping. One such bias, mutual exclusivity, is the assumption that each object has a unique label. When presented with a mix of familiar and unfamiliar objects, children tend to associate novel words with the unknown items. For instance, if a caregiver says, “Look at the blicket,” while the child sees her favorite Ducky, a Mickey Mouse figure, and a novel toy, mutual exclusivity guides the child to infer that “blicket” refers to the toy whose name is unknown. Inspired by such cognitive mechanisms, researchers in computer vision have begun exploring how inductive biases can be incorporated into learning models to improve generalization and interpretability. Among these, shape bias, the tendency to associate category membership with object shape, has received significant attention [ 29 , 18 , 19 ] , particularly due to its relevance in 3D object categorization. In contrast, mutual exclusivity bias, despite being a foundational aspect of human language learning [ 6 , 14 , 15 ] , remains relatively underexplored in computational settings. With the rise of Vision-Language Models (VLMs) that aim to mirror human-like reasoning by grounding language in visual inputs, there is now an opportunity to study mutual exclusivity within a computational framework. These models provide a natural platform for investigating whether and how such biases emerge, especially in complex visual scenes that require reasoning. In this work, we introduce MEBench, a new benchmark for evaluating mutual exclusivity bias in VLMs. Inspired by classic studies in developmental psychology, MEBench formulates the problem as an inference task: given an image containing both known and novel objects, along with a novel label, the model must (1) identify and localize the known categories and (2) associate the novel label with the correct unknown instance by leveraging mutual exclusivity. MEBench goes beyond prior work that focuses solely on object-label association [ 31 ] by introducing realistic, cluttered scenes and additional reasoning requirements. Specifically, models must not only detect and differentiate between known and novel objects but also apply spatial reasoning and contextual cues to resolve ambiguity when multiple unfamiliar items are present. Since no real-world dataset exists for this task, we develop a scalable synthetic data generation pipeline. This pipeline transforms any 3D dataset of object categories into realistic scenes and includes a curated set of procedurally generated novel objects from Thingi10K [ 39 ] , using Blender’s geometry nodes [ 1 ] . To prevent data leakage, novel object labels are sampled from synthetic, non-English-like words (e.g., “dax,” “toma”) commonly used in human studies of mutual exclusivity [ 27 ] . We systematically evaluate state-of-the-art VLMs on MEBench and analyze their performance across multiple subtasks, revealing both capabilities and limitations in their reasoning behavior. Mastery of MEBench would mark a meaningful step toward equipping AI systems with robust zero-shot generalization capabilities, essential for adaptive applications in real-world domains such as home robotics and human-AI interaction. In summary, our contributions are: 1. We introduce MEBench, a computational framework for evaluating mutual exclusivity (ME) bias, and extend its evaluation beyond traditional settings by incorporating spatial reasoning with contextual scene descriptions. 2. We develop a flexible synthetic data generation pipeline that enables controlled experimentation and systematic analysis of factors affecting ME-based learning. 3. We benchmark a diverse set of vision-language models (VLMs) on MEBench, and find that, while these models exhibit weak ME bias, they demonstrate positive spatial reasoning capabilities. 2 Related Work Computational Mutual Exclusivity Modeling. The work most closely related to ours is [ 31 ] , which introduces a generalized low-shot object learning framework that requires applying the mutual exclusivity assumption to associate a novel label with the correct novel object. While our approach also relies on mutual exclusivity, it differs in two key aspects. First, we incorporate a curated set of procedurally generated novel objects, ensuring no data leakage during evaluation. Second, we extend the task beyond traditional mutual exclusivity by introducing a setting with multiple novel objects, where additional reasoning, such as interpreting spatial cues, is required to resolve ambiguity. This level of disambiguation is not addressed in [ 31 ] . Another prior work, [ 6 ] demonstrated that existing feed-forward deep models fail at ME, but did not provide a comprehensive framing or benchmarking SOTA methods. Other recent works have investigated various learning scenarios to explore the strategies children employ when acquiring new concepts [ 9 , 3 , 8 ] . We complement these efforts by introducing a comprehensive benchmark that studies the mutual exclusivity bias commonly observed in infants during the initial stages of word learning. Language-guided Object Detection/Segmentation Tasks. Open-category object detection and segmentation [ 36 , 10 ] have gained traction for their ability to generalize beyond the training vocabulary. These tasks leverage large pre-trained VLMs (e.g. CLIP [ 20 ] ) to detect and segment both known and novel objects—novel to the localization models, though not necessarily to the LLMs—by prompting the LLMs. In contrast, our task focuses on learning object-label mappings, where labels may fall outside typical pretraining vocabularies, such as pseudo-names like “dax” commonly used in psychology studies [ 27 ] . Referring and reasoning segmentation [ 11 , 37 , 22 ] are recent tasks used to evaluate VLMs. In these tasks, models are prompted to segment objects in an image based either on explicit attribute-based descriptions (e.g., “the man in the white shirt") or functional reasoning (e.g., “the object that can be used to eat with"). These settings require VLMs to first comprehend the query, then reason about the object’s attributes or functionality, and finally guide the visual component to segment the target object(s). Our setting extends beyond these tasks by requiring a deeper understanding of the scene, first to visually identify novel objects, then to leverage ME bias to associate a novel label (often a pseudo-name unfamiliar to the model) with the novel object. When multiple novel objects are present, the model must further reason about spatial relationships to resolve ambiguity. Our approach emphasizes inter-object relationships, requiring the model to develop a holistic scene understanding alongside object-centric reasoning. Synthetic 3D Datasets and Generators. Synthetic 3D datasets have been widely used to study various tasks [ 29 , 28 , 34 , 7 , 26 ] , especially when large-scale real-world data is unavailable. Advances in realistic rendering engines [ 7 , 5 , 26 ] and image generation models [ 24 , 25 ] have helped narrow the sim-to-real gap, improving generalization to real-world scenarios. In our work, we use synthetic data to address a novel problem where real-world datasets do not yet exist. This approach enables the creation of diverse, realistic, and controllable environments for scalable experimentation. Vision-Language Detection/Segmentation Models. With advancements in large vision-language models (VLMs), the ability of AI systems to comprehend images and respond to language prompts related to visual content has improved significantly [ 17 , 30 , 4 , 13 , 12 ] Beyond general image understanding, other works [ 33 , 37 , 38 , 22 , 35 ] focus on object grounding, which involves localizing and reasoning about objects within an image and outputting their locations, rather than solely providing text-based responses. These models require a precise scene representation to accurately ground objects based on language prompts. In this work, we use these models as our baselines for our MEBench due to their strong object detection capabilities and their ability to reason effectively with language inputs. Figure 2 : Example of Rendered Data for the MEBench Benchmark. We systematically generate diverse object configurations within varied room backgrounds, ensuring photorealistic renderings that capture realistic spatial arrangements and lighting conditions. 3 MEBench In this section, we first formulate the task in Section 3.1 , followed by a detailed description of the data generation pipeline in Section 3.2 . Finally, we introduce the various data variants used in our study in Section 3.3 . 3.1 Task Formalization Given an RGB image containing multiple objects—some from known everyday categories and one or more novel objects—the model is asked to localize the referent of a novel label (e.g., “Where is the blicket?”). The task involves three subtasks: (1) Object Detection , further divided into: Object Localization —Identifying and localizing all objects present in the scene; and Open-world Recognition. — Differentiating between known and novel instances using visual information; (2) Novel Label Assignment —Applying the mutual exclusivity assumption to correctly associate the novel label with the novel object. When multiple novel objects are present, mutual exclusivity alone is insufficient to resolve ambiguity. To address this, we introduce spatial descriptions (e.g., “The dog is to the right of the blicket”) that encode inter-object relations. The model must then perform: (3) Spatial Reasoning —the ability to interpret spatial descriptions in text to facilitate accurate novel label assignment. Note that this additional input can also be incorporated into the single unknown object scenario. 3.2 Data Generation Pipeline Our system is designed to be compatible with any dataset of 3D object categories. For our benchmark, we use the Toys4K [ 29 ] dataset, chosen for its large number of categories (105) and diverse toy-like object appearances, which closely resemble the variability encountered in real-world child learning scenarios. This dataset comprises common, everyday object categories (e.g., "car", "dog"), which serve as the known categories in our benchmark. Novel Objects. We introduce a curated set of 64 novel objects (see Figure 3 ), sourced from GeoShapeV2 [ 2 ] and Thingi10K [ 39 ] . These objects are manually designed and procedurally generated using geometric nodes in Blender [ 1 ] , ensuring unique and diverse structures. This set is specifically crafted to minimize data leakage, as foundation AI models, trained on vast datasets, have likely encountered most common objects but not such synthetic, abstract, and non-semantic objects. Background. To create diverse and realistic backgrounds, we generate room environments as the backdrop for our scenes. These rooms are primarily living rooms and bedrooms, as they represent the most natural settings for child play. Our room generation is based on Infinigen [ 21 ] , with each room includes varied background object configurations. Lighting conditions are naturally derived from indoor sources or outdoor light filtering through open doors and windows (see Figure 2 ). Data Rendering. During each scene rendering, we first randomly select a subset of objects from the known categories and a set of novel objects from our curated collection. The total number of objects and the number of novel instances are determined based on user-defined input parameters. To achieve realistic object placement, we use rigid body simulation to generate natural rotational poses. Objects are then scaled and positioned at random locations within the scene while ensuring that no two objects collide. Figure 3 : Novel Objects in MEBench. To prevent data leakage during evaluation, we constructed a database of novel objects using procedural generation in Blender [ 1 ] with geometry nodes from GeoShapeV2 [ 2 ] and Thingi10K [ 39 ] . These objects serve as unknown instances, ensuring that models are tested on truly unseen categories. Scene Description Generation. We generate scene descriptions as additional contextual inputs for each view of the scene, rather than for the entire 3D scene. This is because some spatial relationships in 3D are inherently viewpoint-dependent terms like “left," “right," “in front of," and “behind" can vary depending on the observer’s perspective. For each view, we first construct a scene graph using object bounding boxes and the corresponding depth map. In this graph, objects are represented as nodes, while pairwise spatial relationships form the edges. We then translate the structured scene graph representation into plain English descriptions. Please see Sup. for more details. View-point Selection for Inference. During inference, we select rendered viewpoints where all objects in the scene are visible. An object is considered visible in a given view if its segmentation mask occupies at least 200 pixels in a 224\times 224 resolution image. To further minimize variability in object visibility, we run each model three times, each with a different set of selected viewpoints. 3.3 Data Variants To enable a comprehensive analysis and gain deeper insights into the limitations of different methods, we generate datasets with progressively increasing levels of difficulty. Each variant is specifically designed to evaluate three key model capabilities: (1) Object Localization, (2) Mutual Exclusivity (ME) Bias, and (3) Spatial Reasoning. These capabilities are assessed along two primary axes: Number of known objects and Number of novel objects. To maintain clarity, we define data variant acronyms based on these axes. For example, 1K-0U represents a scenario with 1 known object and 0 novel objects. These data variants provide a framework for systematically testing model capabilities, distinguishing basic object detection from higher-level reasoning tasks such as ME bias and spatial inference. Object Localization: Evaluating Detection Ability. The first group consists of only one known object, without any novel objects: 1K-0U. This setting assesses the model’s ability to localize known objects. A model that performs well in this group demonstrates object detection capabilities, independent of novel object inference. Mutual Exclusivity Bias: Assigning Novel Labels Correctly. The second group introduces one novel object alongside known objects (1K-1U and 2K-1U). This group evaluates the model’s ability to apply the ME assumption, which assigns the novel label to the novel object. Spatial Reasoning: Disambiguating Multiple Novel Objects. The third group contains two novel objects: 1K-2U, introducing inherent ambiguity that cannot be resolved solely through ME bias. This setting evaluates the model’s ability to leverage spatial descriptions to disambiguate novel objects based on their relationships within the scene. 4 Experiments 4.1 Baselines We evaluate our MEBench benchmark on seven SOTA VLM baselines, consisting of both closed-source and open-source models: CogVLM [ 33 ] , Gemini [ 30 ] , Sa2VA [ 37 ] , OMG-LLaVA [ 38 ] , LISA [ 11 ] , LLaVA-OV [ 12 ] , and F-LMM [ 35 ] . The majority of these VLMs are trained primarily for object grounding—performing object detection or segmentation based on text prompts. In contrast, LLaVA-OV [ 12 ] is designed as a more general VQA and reasoning model. The diversity in model architectures and training objectives allows us to conduct a comprehensive analysis, gaining deeper insights into how different VLMs perform on our benchmark across various reasoning and perception challenges. 4.2 Data Using the data generation pipeline from Section 3.2 , we generate 100 scenes per setting. with 25 randomly sampled viewpoints per scene. During inference, we ensure that all objects are visible in the selected views to maintain consistency and fairness in evaluation (see Section 3.2 ). 4.3 Evaluation Protocol & Metrics In this subsection, we outline the evaluation criteria for each subtask and the metrics used to assess model performance. Object Localization. We assess the models’ ability to detect objects using the standard Average Precision at IoU threshold ( AP@t ) metric, commonly used in object detection tasks [ 23 , 32 ] . Mutual Exclusivity Bias. We define p(x_{n\rightarrow n}) as the probability of correctly assigning the novel label to a novel object, equivalent to the AP@t metric for novel objects and the ME score from [ 6 ] . Similarly, p(x_{k\rightarrow k}) denotes correct assignment of known labels, and p(x_{n\rightarrow k}) captures incorrect assignment of the novel label to a known object. (a) (b) (c) Figure 4 : Object Detection Performance of VLMs on Known Objects in the ( 4(a) ) 1K-0U (1 known and 0 unknown object), ( 4(b) ) 1K-1U (1 known and 1 unknown objects), and ( 4(c) ) 2K-1U (2 known and 1 unknown objects). To evaluate ME bias, we focus on conditional metrics given correct known-object classification: p(x_{n\rightarrow n}|x_{k\rightarrow k}) : correct novel label assignment; and p(x_{n\rightarrow k}|x_{k\rightarrow k}) : incorrect novel label assignment to a known object. An effective model should maximize p(x_{n\rightarrow n}|x_{k\rightarrow k}) while minimizing p(x_{n\rightarrow k}|x_{k\rightarrow k}) . We capture this trade-off using a normalized ME score: -1\leq\text{ME}=\frac{p(x_{n\rightarrow n}|x_{k\rightarrow k})-p(x_{n\rightarrow k}|x_{k\rightarrow k})}{p(x_{n\rightarrow n}|x_{k\rightarrow k})+p(x_{n\rightarrow k}|x_{k\rightarrow k})}\leq 1 We focus on the posterior term x_{k\rightarrow k} , as correct classification of known objects should lead a model with ME bias to assign the novel label to the novel object. The ME score is defined as: (1) -1\leq ME\leq 0 : Tendency to misassign the novel label to a known object, which indicates weak ME bias; (2) 0