What’s in Common? Multimodal Models Hallucinate When Reasoning Across Scenes

Candace Ross, Florian Bordes, Adina Williams, Polina Kirichenko, Mark Ibrahim · 2025

Read What’s in Common? Multimodal Models Hallucinate When Reasoning Across Scenes by Candace Ross, Florian Bordes, Adina Williams, Polina Kirichenko, Mark Ibrahim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03768.txt.

Introduction

Candace Ross and colleagues build Common-O, a benchmark that catches multimodal models hallucinating when they reason across scenes. Models name objects in single images impressively, yet real-world reasoning still trips them into inventing content. Common-O asks a deceptively simple question across fresh, never-web-published photos: what is in common?

More than ten thousand examples, modeled on human cognitive tests, force comparison rather than mere labeling. Leading models, including ones trained for step-by-step reasoning, stumble well below their saturated perception scores. The gap shows that spotting objects is not the same as reasoning about them jointly. A contamination-free set gives the field an honest ruler for scene reasoning progress.

Catalog record

CREATORCandace Ross, Florian Bordes, Adina Williams, Polina Kirichenko, Mark Ibrahim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03768

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What’s in Common? Multimodal Models Hallucinate When Reasoning Across Scenes Candace Ross, Florian Bordes, Adina Williams, Polina Kirichenko, Mark Ibrahim Abstract Multimodal language models possess a remarkable ability to handle an open-vocabulary’s worth of objects. Yet the best models still suffer from hallucinations when reasoning about scenes in the real world, revealing a gap between their seemingly strong performance on existing perception benchmarks that are saturating and their reasoning in the real world. To address this gap, we build a novel benchmark of in-the-wild scenes that we call Common-O Bench . With more than 10.5k examples using exclusively new images not found in web training data to avoid contamination, Common-O Bench goes beyond just perception, inspired by cognitive tests for humans, to probe reasoning across scenes by asking “what’s in common?”. We evaluate leading multimodal language models, including models specifically trained to perform chain-of-thought reasoning. We find that perceiving objects in single images is tractable for most models, yet reasoning across scenes is very challenging even for the best models, including reasoning models. Despite saturating many leaderboards focusing on perception, the best performing model only achieves 35% on Common-O Bench —and on Common-O Complex , consisting of more complex scenes, the best model achieves only 1%. Curiously, we find models are more prone to hallucinate when similar objects are present in the scene, suggesting models may be relying on object co-occurrence seen during training. Among the models we evaluated, we found scale can provide modest improvements while models explicitly trained with multi-image inputs show bigger improvements, suggesting scaled multi-image training may offer promise. We make our benchmark publicly available to spur research into the challenge of hallucination when reasoning across scenes. † † affiliation: FAIR at Meta † † date: August 10, 2026 † † HuggingFace Dataset: https://huggingface.co/datasets/facebook/Common-O 1 Introduction Figure 1: Reasoning across scenes is an open challenge for today’s best multimodal models. We show the best performance from the Open VLM leaderboard on MMBench and single image evaluations from our benchmark illustrating saturation for perception tasks. Multimodal models today are starting to saturate visual perception leaderboards. For example, on classical text-and-image benchmarks such as CLEVR ( Johnson et al. 2017 ) , DocVQA ( Mathew et al. 2021 ) , ChartQA ( Masry et al. 2022 ) , TextVQA ( Singh et al. 2019 ) , MMBench ( Liu et al. 2023 ) , and Seed-Bench ( Li et al. 2024b ) , top-performing models achieve an accuracy of 80%-90% ( Zhang et al. 2024 ) . However, despite these impressive results, there is a growing concern that these benchmarks may not accurately reflect the performance of models in real-world settings. In fact, research has shown that models often struggle to generalize to new, unseen data, and are prone to hallucinating objects that are not present in the scene ( Guan et al. 2023 ) . One of the main reasons for this gap between benchmark performance and real-world performance is the contamination between benchmarks and training data ( Chen et al. 2024 ) . Many benchmarks are constructed from web data, which is also used to train models. This means that models are often trained on data that is quite similar to the benchmark data, leading to an overestimate of their performance. Another factor contributing to the gap is the lack of benchmarks that capture the complexity and variability of real-world scenes. Many benchmarks use simplified geometric visuals or static scenes; while useful for static tasks such as chart understanding, these do not reflect the dynamic and evolving nature of real-world scenes. The ability to reason across complex scenes containing multiple objects is a fundamental aspect of human cognition. Research in cognitive science has shown object identification in scenes is a key component of cognitive function—and that deficits in this ability are a hallmark of cognitive decline, such as in Alzheimer’s disease ( Takechi and Dodge 2010 ) . Furthermore, studies have demonstrated that the brain’s ability to understand the relationships between objects in a scene is closely tied to its ability to understand the scene as a whole ( Damasio et al. 2001 ; Brandman and Peelen 2017 ) . These findings suggest that a benchmark that requires models to reason about complex, dynamic scenes is crucial step towards deploying reliable models in the real world. To address these challenges, we introduce a new multi-image benchmark, Common-O Bench , designed to test models’ ability to reason across dynamic scenes in a way more similar to human reasoning. Our benchmark includes scenes containing multiple objects with varying lighting conditions, and complex backgrounds that requires models to reason about the relationships between objects across distinct scenes. We choose scenes with up to 7 objects as our default setting, inspired by the 1956 classic, putative, heuristic constraint on human memory, colloquially described as the “Magical Number Seven, Plus or Minus Two” ( Miller 1956 ; Baddeley 1994 ; Cowan et al. 2007 , i.a.). Common-O Bench comprises both real and synthetic data, allowing for more flexibility in our evaluation, as we can sample a wide range of object-background combinations that are typical in real data. We also provide a non-overlapping fully synthetic challenge set, Common-O Complex that spans up to 16 objects per scene, increasing scene complexity appreciably. In both Common-O Bench and Common-O Complex , we intentionally provide multiple camera points of view of a given scene, reflecting the diversity found in the real world. We find that despite being able to recognize objects in individual scenes, state-of-the-art models struggle to reason across scenes. The best performing model we tested, GPT-4o, achieves only 35% on Common-O Bench , highlighting reasoning across scenes as open challenge, in stark contrast to the saturation observed for other multimodal benchmarks. For the more challenging set Common-O Complex , the best performing model achieve < 1%. Curiously, we find models hallucination is pervasive , with at least one 1 object hallucinated 53% of the time and 2+ as often as 23% of the time. Our findings have important implications for the development of multimodal models ( Bordes et al. 2024 ) . We find models trained with multi-image inputs achieve higher performance and scale can yield marginal benefits, yet even the best multi-image large scale models struggle highlighting the need for models to be designed with real-world scenes in mind. This requires a fundamental shift in the way models are designed and trained, and underscores the need for more research in this area. We release Common-O Bench and Common-O Complex to mark a new challenge in multimodal models’ ability to reason across scenes that we hope could unlock new frontiers in real world applications 1 1 1 Datasets are available at https://huggingface.co/datasets/facebook/Common-O . 2 Related Work Many works have aimed to evaluate model performance on visual reasoning. We summarize our contributions relative to existing benchmarks in terms of multi-image capability, scale, and saturation in Table 1 . Our dataset is larger in size, captures multi-image reasoning across scenes inspired by human cognitive tests, and stands out in terms of not relying on existing web datasets, thereby avoiding possible training data contamination or object resemblance. Together, these factors make our benchmark much more challenging

Read the complete text