A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection
Kunal Runwal, Swaraj Gajare, Daniel Adejumo, Omkar Ankalkope, Siddhant Baroth, Aliasghar Arab · 2026
Read A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection by Kunal Runwal, Swaraj Gajare, Daniel Adejumo, Omkar Ankalkope, Siddhant Baroth, Aliasghar Arab (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28888.txt.
Introduction
Runwal and co-authors test a semantic observer layer for self-driving cars. Some hazards depend on context and escape pixel level detectors. Their layer runs a quantized vision-language model at 1 to 2 hertz beside the main control loop. It watches for semantic edge cases and triggers fail-safe handoffs. With NVFP4 quantization and FlashAttention2, inference took about 500 milliseconds on their hardware. That is a 50 fold speedup over the unoptimized baseline. They also flag an NF4 recall collapse as a hard deployment limit and map metrics to safety goals.
Catalog record
| CREATOR | Kunal Runwal, Swaraj Gajare, Daniel Adejumo, Omkar Ankalkope, Siddhant Baroth, Aliasghar Arab |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34691 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28888
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Semantic Observer Layer for Autonomous Vehicles: Pre-Deployment Feasibility Study of VLMs for Low-Latency Anomaly Detection Kunal Runwal, Swaraj Gajare, Daniel Adejumo, Omkar Ankalkope, Siddhant Baroth, Aliasghar Arab Abstract Semantic anomalies—context-dependent hazards that pixel-level detectors cannot reason about—pose a critical safety risk in autonomous driving. We propose a semantic observer layer : a quantized vision-language model (VLM) running at 1–2 Hz alongside the primary AV control loop, monitoring for semantic edge cases, and triggering fail-safe handoffs when detected. Using Nvidia Cosmos-Reason1-7B with NVFP4 quantization and FlashAttention2, we achieve ∼ \sim 500 ms inference—a ∼ \sim 50 × \times speedup over the unoptimized FP16 baseline (no quantization, standard PyTorch attention) on the same hardware—satisfying the observer timing budget. We benchmark accuracy, latency, and quantization behavior in static and video conditions, identify NF4 recall collapse (10.6%) as a hard deployment constraint, and a hazard analysis mapping performance metrics to safety goals. The results establish a pre-deployment feasibility case for the semantic observer architecture on embodied-AI AV platforms. I INTRODUCTION I-A Motivation Undetected semantic anomalies are a direct hazard in autonomous driving: an AV that cannot distinguish a deflated ball on the road from a shadow, or misreads traffic lights on a transport truck, may take incorrect or fatal actions [ 20 , 18 , 22 ] . The broader robotics community has recognized the same need—recent work on spatial awareness [ 14 ] , traversability [ 10 , 9 ] , and embodied navigation [ 6 ] all confirm that safe operation in unstructured outdoor environments demands context-aware semantic reasoning, not just pixel-level detection. Yet none of these systems deliver this reasoning within the sub-second latency budget required for on-vehicle deployment. Building a framework that is both semantically capable and fast enough to act in time is therefore the central challenge this work addresses. I-B Problem Statement Fig. 1 : Qualitative results on the Hazard Perception Test Dataset [ 17 ] using Cosmos-Reason1-7B [ 13 ] . Top row (Samples 11–12): correctly classified normal frames. Bottom row (Samples 13–14): a normal frame and a detected anomaly, demonstrating context-aware semantic reasoning. Current AV anomaly detectors are brittle to out-of-distribution objects because they lack semantic and temporal reasoning [ 7 ] . LLM-based approaches recover this reasoning capability but are too slow for on-vehicle deployment, and pipelines that chain a visual parser with a separate LLM compound errors at the vision-language interface [ 16 , 7 ] . Vision-Language Models (VLMs) address the alignment problem and have been applied to traffic anomaly resolution [ 15 ] and multi-agent interaction reasoning [ 11 ] , yet they inherit the same latency barrier. What is missing is a deployable framework that integrates VLM-level semantic reasoning within the timing budget of a real AV platform—which is precisely what this work builds and benchmarks. We address these limitations by studying a unique integration of quantized VLMs and attention-efficient inference kernels targeted at an embodied-AI autonomous vehicle test-bed at NYU. The proposed architecture introduces an observer-level semantic monitor that operates alongside—not inside—the primary AV control loop. As illustrated in Fig. 2 , this observer layer sits between the regular autonomy stack and the fail-safe stack, acting as an orchestrator : running at 1–2 Hz, continuously monitoring for semantic edge cases the primary stack cannot reason about, and triggering a graceful handoff to the fail-safe stack upon high-confidence anomaly detection. Because the observer is not in the critical control path, its 500 ms inference budget is acceptable—semantic anomalies evolve over seconds, not milliseconds—and high precision (minimizing spurious fail-safe activations) is the constraint of the operation design. This paper proposes and formalizes the semantic observer as a distinct autonomy layer—running at 1–2 Hz between the regular autonomy stack and the fail-safe stack—responsible for detecting semantic edge cases and orchestrating fail-safe handoffs, decoupling high-level scene reasoning from the primary control loop. We provide the first systematic evaluation of Cosmos-Reason1-7B as a semantic observer, characterizing accuracy, latency, and quantization behavior across static image and video conditions on three public datasets. A key negative finding is that NF4 4-bit quantization causes catastrophic recall collapse (10.6%) under video inference while performing competitively on static images, establishing a hard per-modality deployment constraint with actionable quantization selection guidance for safety-critical AV systems. Fig. 2 : Semantic observer layer architecture. The VLM observer runs at 1–2 Hz alongside the primary AV control loop, processing temporal windows of RGB frames with a structured prompt. Upon detecting a high-confidence semantic constraint violation, it triggers a fail-safe handoff. Visual tokens from Cosmos-Reason1-7B are projected into the language embedding space and evaluated against context-conditioned semantic constraints to produce a binary {Normal, Anomaly} decision. II PROBLEM FORMULATION To define anomalies in a mathematically principled manner, we formalize the system in terms of the tasks performed by a VLM to produce external anomaly decisions. The system operates on forward-facing RGB video frames from a vehicle in motion, processed in temporally batched windows, together with a structured text prompt that describes semantic constraints and the desired output format. The prompt is designed to elicit context-aware reasoning over scene elements and their consistency with nominal driving behavior. The output is a binary class \text{\lx@text@lbrace normal},\text{anomaly\lx@text@rbrace} . This is the chosen format because we prefer an incremental approach rather than quantifying a severity score. For future implementations and adaptations such as anomaly mitigation, severity/risk scores would help shape control strategies. II-A System Definition We represent the external anomaly detection system as a tuple T_{\mathrm{ext}}=\{S,O_{\mathrm{rgb}},X,U,Z,\Phi,\varepsilon\}, (1) where S denotes the ego-vehicle state space, O_{\mathrm{rgb}} denotes RGB visual observations, X denotes contextual information, U denotes predictive uncertainty, Z denotes the binary external anomaly output, \Phi denotes explanation artifacts, and \varepsilon denotes system effectiveness metrics. II-B Video Representation Let V be an RGB video consisting of F frames: V=\{I_{i}\}_{i=1}^{F},\quad I_{i}\in O_{\mathrm{rgb}}, (2) where I_{i} is the i -th frame of the video. Each frame is temporally aligned with an ego state s_{i}\in S and contextual information x\in X, which may include road geometry, intersection topology, traffic rules, and route intent. II-C Semantic Scene Interpretation Given a temporal window of frames I_{i-k:i} , a vision-language model maps the visual input and a fixed prompt \mathcal{P} to a semantic scene representation f_{\mathrm{VLM}}:O_{\mathrm{rgb}}^{k+1}\times\mathcal{P}\rightarrow\mathcal{R} (3) \hat{r}_{I_{i-k:i}}=f_{\mathrm{VLM}}(I_{i-k:i},\mathcal{P}), (4) where \hat{r}_{I_{i-k:i}}\in\mathcal{R} encodes inferred objects, agents, traffic signals, and scene attributes. The prompt \mathcal{P} conditions the model to reason about safety-relevant and context-dependent anomalies. II-D Contextual Semantic Constraints Let C(x(\mathcal{P}),\hat{r}_{I_{i-k:i}}) denote a set of semantic constraints defining nominal external behavior under context x\triangleq x(\mathcal{P})\in X , as interpreted through the prompt \mathcal{P} . These constraints encode expectations such as: • absence of unexpected objects in drivable regions, • consistency of traffic-light states with intersection semantics, • static behavior of road signs and infrastructure, • compatibility of observed agent behavior with traffic rules. II-E Constraint Violation and Binary Anomaly Output We define a semantic violation operator as Equation 5 which evaluates whether the inferred semantics \hat{r}_{I_{i-k:i}} violate any constraint in C(x(\mathcal{P}),\hat{r}_{I_{i-k:i}}) . The operator returns a non-negative value indicating the presence of a violation. The binary external anomaly output is defined by Equation 6 . v_{I_{i-k:i}}=\mathrm{Viol}\big(C(x(\mathcal{P}),\hat{r}_{I_{i-k:i}})\big), (5) z_{I}=\begin{cases}1,&\text{if }v_{I}>0,\\ 0,&\text{otherwise}.\end{cases} (6) Z=(z_{I_{1}},z_{I_{2}},\dots,z_{I_{\frac{F}{k}}}),\quad Z_{I}\in\{0,1\}, (7) Collectively, the system output over the video is defined by Equation 7 where each