Vision Foundation Model Embedding-Based Semantic Anomaly Detection
Max Peter Ronecker, Matthew Foutter, Amine Elhafsi, Daniele Gammelli, Ihor Barakaiev, Marco Pavone, Daniel Watzenig · 2025
Read Vision Foundation Model Embedding-Based Semantic Anomaly Detection by Max Peter Ronecker, Matthew Foutter, Amine Elhafsi, Daniele Gammelli, Ihor Barakaiev, Marco Pavone, Daniel Watzenig (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07998.txt.
Introduction
Max Peter Ronecker leads a team of seven authors in 2025 in catching semantic anomalies, scenes where familiar objects combine in ways that should never happen. A stop sign on a kitchen table is made of ordinary parts, yet the whole is wrong, and such scenes can send an autonomous system into undefined behavior.
Their detector leans on vision foundation models, huge networks trained on general images. It compares local features of a live image against a library of safe, nominal scenes stored in advance. Two variants of the framework are tested. The approach aims at robots and vehicles that must notice nonsense, not just novelty.
Catalog record
| CREATOR | Max Peter Ronecker, Matthew Foutter, Amine Elhafsi, Daniele Gammelli, Ihor Barakaiev, Marco Pavone, Daniel Watzenig |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07998
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Vision Foundation Model Embedding-Based Semantic Anomaly Detection Max Peter Ronecker, Matthew Foutter, Amine Elhafsi, Daniele Gammelli, Ihor Barakaiev, Marco Pavone, Daniel Watzenig Abstract Semantic anomalies are contextually invalid or unusual combinations of familiar visual elements that can cause undefined behavior and failures in system-level reasoning for autonomous systems. This work explores semantic anomaly detection by leveraging the semantic priors of state-of-the-art vision foundation models, operating directly on the image. We propose a framework that compares local vision embeddings from runtime images to a database of nominal scenarios in which the autonomous system is deemed safe and performant. In this work, we consider two variants of the proposed framework: one using raw grid-based embeddings, and another leveraging instance segmentation for object-centric representations. To further improve robustness, we introduce a simple filtering mechanism to suppress false positives. Our evaluations on CARLA-simulated anomalies show that the instance-based method with filtering achieves performance comparable to GPT-4o, while providing precise anomaly localization. These results highlight the potential utility of vision embeddings from foundation models for real-time anomaly detection in autonomous systems. I INTRODUCTION Autonomous vehicles, such as Waymo [ 1 ] or Tesla [ 2 ] , are increasingly deployed in real-world environments and rely heavily on machine learning (ML) algorithms, especially in perception modules, e.g., object detection. While these algorithms often perform reliably within their training distributions, ML models remain vulnerable to out-of-distribution (OOD) inputs, which can lead to unsafe or unpredictable behavior. An OOD input is data that significantly differs from the training distribution of an ML model and is defined relative to that model, such as unusual objects, rare weather conditions, or novel environments. A common mitigation strategy to avoid unpredictable failure modes involves detecting OOD situations at runtime and transitioning the system to a safe state [ 3 ] . Among OOD observations, semantic anomalies pose a unique challenge. As defined in [ 4 ] , semantic anomalies reflect failures in high-level reasoning rather than low-level perception or control. In contrast to traditional OOD cases, semantic anomalies are defined with respect to the system’s context and operational domain and involve in-distribution elements arranged in atypical or contextually invalid ways—for example, a stop sign on a billboard or a traffic light mounted on a moving truck may confuse an autonomous vehicle, despite all visual components being familiar. These examples are inspired by real-life occurrences, such as those observed in the Tesla system [ 5 , 6 ] shown in Fig. 1 . These anomalies are often trivial for humans to interpret but difficult for conventional ML models to detect. The emergence of Large Language Models (LLMs) and Vision-Language Models (VLMs) provides new tools for addressing this gap, leveraging their strong reasoning and generalization capabilities for semantic anomaly detection using textual prompts, multi-modal embeddings, or direct image-based reasoning [ 4 , 7 ] . While foundation models offer zero-shot capabilities [ 8 , 9 , 10 ] and perform well on nominal data, these high-capacity ML models often suffer from high latency—with response times of several seconds—and are prone to hallucination [ 11 ] . Embedding-based methods are faster but typically provide only a coarse anomaly score without spatial localization [ 7 ] . This motivates the development of efficient, embedding-based approaches that can both detect and localize semantic anomalies in real time. In this work, we take a first step toward this goal. Fig. 1 : Examples of semantic anomalies (a,c) and their CARLA-simulated equivalents (b,d). A truck with traffic lights (a) confused a Tesla into detecting active signals [ 5 ] . A stop sign on a billboard (c) caused unintended braking [ 12 , 6 ] . We evaluate existing semantic anomaly detectors and propose a vision embedding-based framework for anomaly detection and localization. The framework is tested on simulated CARLA data, following [ 4 , 7 ] . Our contributions are as follows: • Evaluation of existing VLM-based semantic anomaly detectors: We provide a detailed evaluation of existing VLM-based semantic anomaly detectors at the frame level, identifying their strengths and weaknesses. • Embedding-based semantic anomaly detection: We propose two variations of a foundation model embedding-based semantic anomaly detection framework that operates directly on images. Our approach achieves detection performance comparable to large vision-language models such as GPT-4o, while also enabling precise localization of anomalies. Extensive evaluation on simulated data highlights the potential of vision embeddings for semantic anomaly detection. • Filtering techniques for embedding-based semantic anomaly detection: We introduce a simple yet effective filtering technique that further boosts the performance of our embedding-based framework, helping it reach GPT-4o-level results. II RELATED WORK II-A Vision Foundation Models Foundation models are large-scale neural networks trained on broad, often internet-scale data to perform well across many tasks with minimal adaptation [ 13 ] . Models like CLIP [ 14 ] , DINO [ 15 ] , and DINOv2 [ 10 , 16 ] have demonstrated the capability of large-scale pretraining for learning general-purpose visual features. CLIP learns joint image-text embeddings from 400 million image-caption pairs, enabling zero-shot classification across diverse categories. DINO introduced self-supervised learning using Vision Transformers (ViTs), showing that meaningful object-centric representations can emerge without labels. DINOv2 builds on this by training on 142 million curated images, yielding robust and transferable features across a wide range of visual tasks (e.g., image and instance recognition) with minimal fine-tuning. Segment Anything [ 17 ] and its follow-up model [ 18 ] introduce general-purpose segmentation models trained on large-scale instance mask datasets. These models enable automatic or promptable segmentation of arbitrary objects, using object detectors as prompts [ 19 , 20 ] . Combined, these models can potentially be used to provide semantically rich embeddings that can support anomaly detection by identifying distributional shifts or unexpected objects in the scene. II-B Foundation Models for Anomaly Detection in Robotics Semantic anomalies, as defined in [ 4 ] refer to failures in high-level reasoning rather than low-level perception or control. Examples include autonomous vehicles braking for stop signs on billboards or failing to interpret unusual traffic configurations. A proposed approach leverages large language models (LLMs) to monitor a robot’s perceptions and decisions, flagging behaviors that deviate from human intuition. Experiments in driving and manipulation tasks show that such LLM-based monitors align well with human judgment. Sinha et al. [ 7 ] introduce a two-stage system: a lightweight classifier detects anomalies from LLM/VLM embeddings, triggering a slower generative LLM to reason and suggest recovery. Coupled with model-predictive control, this enables safe replanning. The fast detector outperforms GPT-4 in failure detection, showing embeddings suffice for real-time use. Other foundation model-based approaches for anomaly and OOD detection include S2M [ 21 ] , which converts anomaly scores into segmentation masks by generating box prompts for the segmentation model SAM, and AnomalyCLIP [ 22 ] , a zero-shot method that learns object-agnostic prompts and applies a glocal loss for accurate detection and segmentation without target domain data. However, these methods either require large models and significant compute, making real-time processing difficult, fail to localize the anomaly, or are not suited for complex semantic anomalies. The proposed method aims to provide a lightweight, embedding-based approach capable of detecting and localizing semantic anomalies. III METHODOLOGY Fig. 2 : Overview of the