CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environments

Rishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, Antoine Bosselut · 2025

Read CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environments by Rishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, Antoine Bosselut (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26006.txt.

Introduction

Rishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, and Antoine Bosselut introduce a benchmark for spotting and explaining odd things in real images. People easily notice, reason about, and explain anomalies in a scene. Computer vision work has mostly used factory defects or synthetic fakes, which miss the range and surprise of real world anomalies.

Their benchmark, CAVE, supports three open ended tasks: describing an anomaly, explaining it, and justifying why it counts as one. Detailed annotations mark where the anomaly sits and rate its complexity, severity, and commonness, drawing on cognitive science. Top vision language models still struggle with both perception and commonsense reasoning on the tasks, even with advanced prompting.

Catalog record

CREATORRishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, Antoine Bosselut
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26006

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environments Rishika Bhagwatkar, Syrielle Montariol, Angelika Romanou, Beatriz Borges, Irina Rish, Antoine Bosselut Abstract Humans can naturally identify, reason about, and explain anomalies in their environment. In computer vision, this long-standing challenge remains limited to industrial defects or unrealistic, synthetically generated anomalies, failing to capture the richness and unpredictability of real-world anomalies. In this work, we introduce CAVE, the first benchmark of real-world visual anomalies. CAVE supports three open-ended tasks: anomaly description, explanation, and justification; with fine-grained annotations for visual grounding and categorizing anomalies based on their visual manifestations, their complexity, severity, and commonness. These annotations draw inspiration from cognitive science research on how humans identify and resolve anomalies, providing a comprehensive framework for evaluating Vision-Language Models (VLMs) in detecting and understanding anomalies. We show that state-of-the-art VLMs struggle with visual anomaly perception and commonsense reasoning, even with advanced prompting strategies. By offering a realistic and cognitively grounded benchmark, CAVE serves as a valuable resource for advancing research in anomaly detection and commonsense reasoning in VLMs. We release the code and benchmark on our project webpage. 1 1 1 https://smontariol.github.io/cave-visual-anomalies/ * * footnotetext: Equal contribution. 1 Introduction “If you notice an abnormal situation, please contact an agent.” Such announcements are commonplace in public spaces worldwide, highlighting a fundamental human trait: the ability to detect anomalies. Identifying uncommon situations, behaviors, and other elements that deviate noticeably from a norm is a natural and expected behavior for humans Klein et al. 2007 ; Klein 2013 . Figure 1: CAVE Example. CAVE contains images captured in real-world scenarios, annotated with commonsense anomalies, along with their textual explanations, justifications and bounding boxes around anomalous elements. It also includes numerical features representing how humans perceive these anomalies. As Vision-Language Models’ (VLMs) Li et al. 2024a ; OpenAI 2024 ; Awadalla et al. 2023 ; Laurençon et al. 2024 functionality broadens and deployment in real-world scenarios expands Jin et al. 2024 ; Xu et al. 2024 , so does their exposure to unexpected and novel situations. In this expanding landscape, their ability to differentiate between normal and anomalous situations is crucial for ensuring safe and efficient operation Mullen Jr et al. 2024 . These models leverage the comprehensive world knowledge and reasoning capabilities of their Large Language Model (LLM) backbones ( McKinzie et al. 2024 ; Liu et al. 2024a ; Karamcheti et al. 2024 ; Laurençon et al. 2024 ) , equipping them to handle a variety of tasks Liu et al. 2024a ; Caffagni et al. 2024 . However, rare or uncommon situations are inherently underrepresented in training data, making it challenging for models to learn how to recognize them and to react to them. Moreover, VLMs tend to hallucinate image content toward the most probable interpretation, which directly conflicts with their ability to identify unexpected situations Zhou et al. 2023 . While it is crucial to accurately evaluate VLM’s ability to identify and understand anomalous situations, the scope of existing visual anomaly detection benchmarks is limited. In the literature, visual anomaly detection is mainly applied to specific domains such as industrial inspection Chandola et al. 2009 ; Diers and Pigorsch 2023 ; Bergmann et al. 2019a ; Xie et al. 2024 , medical diagnosis Fernando et al. 2021 ; Zhang et al. 2020 or video surveillance Sultani et al. 2018 . More recently, commonsense-oriented anomaly detection benchmarks have started to appear. They typically rely on synthetic image generation to create artificial scenarios Bitton-Guetta et al. 2024 ; Li et al. 2024b ; Roman and Meyer 2024 ; Bitton-Guetta et al. 2023 ; Tai et al. 2024 . Non-synthetic approaches rely on domain-specific datasets, such as understanding creative elements in advertisements Malakouti et al. 2024 or detecting video game glitches Taesiri et al. 2024 . As a result, existing benchmarks fail to capture the diversity, unpredictability, and realism of real-world anomalies, leaving a critical gap in the evaluation of VLMs’ true anomaly detection capabilities. In this work, we introduce C ommonsense A nomalies in V isual E nvironments (CAVE), the first visual anomaly benchmark curated from images captured from a human perspective, in real-life settings or as screenshots from smartphones and laptops. Building on top of cognitive science literature, we pair each extracted anomalous image with annotations supporting three open-ended tasks that align with human anomaly detection and sense-making processes: anomaly description, anomaly explanation, and anomaly justification. Additionally, we support an anomaly localization task to evaluate the visual grounding capabilities of the models. We also categorize anomalies based on the type of visual reasoning required to identify them ( e.g. , spatial or attribute reasoning) and further label them with three numerical features: (a) how severe an anomaly is, (b) how surprising and uncommon it is, and (c) how complex it is to detect (see Figure 1 ). Our main contributions and findings are as follows : • We propose an anomaly understanding framework that builds upon cognitive science literature regarding the way humans identify and understand anomalies ( Section 2 ). We split the detection process into three sub-tasks formalized as open-ended visual question answering, and include a classification system based on visual manifestations and numerical attributes. This novel framework allows for systematic characterization and annotation of visual commonsense anomalies. • We introduce CAVE, a benchmark curated from Reddit comprising 361 images designed to evaluate VLMs’ ability to detect and understand anomalies ( Section 3 ). It captures a wide range of anomalies varying in visual manifestation, commonness, severity, and complexity. • We evaluate 3 proprietary models and 5 open-source state-of-the-art models on CAVE, experimenting with 5 advanced prompting strategies ( Section 4 ). We show that the best model, GPT-4o, only reaches 57% F1-score on anomaly detection with a multi-step reasoning strategy, highlighting significant room for improvement. • We analyse VLMs’ success and failure modes, finding that they perform better on surprising and severe anomalies but struggle with anomalies involving complex visual perception abilities, especially spatial reasoning and pattern detection. 2 Theoretical Framework We leverage cognitive science literature to formalize the way humans detect and understand anomalies into a set of tasks. This framework guides our dataset creation process, model assessment, and analysis, allowing us to explore the alignment between human and machine processing of visual anomalies. 2.1 Perception of the anomaly Anomaly detection focuses on identifying deviations from expected patterns Klein et al. 2007 ; Klein 2013 . In this work, we define an anomaly not simply as a statistical rarity Grubbs 1969 ; Chandola et al. 2009 ; Pang et al. 2021 but as a situation that disrupts an established pattern or expectation. This perspective underscores the key human ability to construct mental models of the world and identify deviations from these models Klein et al. 2005 . Mental models are cognitive representations of the world, guiding information processing and anticipation of events Borders et al. 2024 . When an individual encounters something unexpected or surprising that disrupts their established mental model, it can be perceived as an anomaly Klein et al. 2007 ; however, humans’ accurate perception of situations is limited when doing rapid visual processing Treisman and Schmidt 1982 ; De Keyser and Woods 1990 . The process of identifying

Read the complete text