Cerberus: Real-Time Video Anomaly Detection via Cascaded Vision-Language Models

Yue Zheng 1 , Xiufang Shi 1 , Jiming Chen 2, 3 , Yuanchao Shu 2,† · 2025

Read Cerberus: Real-Time Video Anomaly Detection via Cascaded Vision-Language Models by Yue Zheng 1 , Xiufang Shi 1 , Jiming Chen 2, 3 , Yuanchao Shu 2,† (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16290.txt.

Introduction

Yue Zheng and three co-authors build Cerberus, a fast guard for spotting odd events in video feeds. Vision-language models can spot trouble in new settings with no retraining. But they run too slowly and ground their gaze poorly for live use. Cerberus splits the job: a light filter screens frames, and the big model reasons only over flagged ones.

Two ideas drive the gains. Motion mask prompting steers the big model's eye toward the parts of the scene that move. Rule-based checks flag acts that break learned norms of normal behavior, instead of listing every odd thing that could occur. Across four data sets, Cerberus runs at about fifty-eight frames per second. That is more than one hundred fifty times faster than plain model use. Its accuracy stays near the best reported, making live video watch with smart models practical.

Catalog record

CREATORYue Zheng 1 , Xiufang Shi 1 , Jiming Chen 2, 3 , Yuanchao Shu 2,†
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16290

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cerberus: Real-Time Video Anomaly Detection via Cascaded Vision-Language Models Thanks: † Corresponding author: Yuanchao Shu ([email protected]). Yue Zheng 1 , Xiufang Shi 1 , Jiming Chen 2, 3 , Yuanchao Shu 2,† Abstract . Video anomaly detection (VAD) has rapidly advanced by recent development of Vision-Language Models (VLMs). While these models offer superior zero-shot detection capabilities, their immense computational cost and unstable visual grounding performance hinder real-time deployment. To overcome these challenges, we introduce Cerberus , a two-stage cascaded system designed for efficient yet accurate real-time VAD. Cerberus learns normal behavioral rules offline, and combines lightweight filtering with fine-grained VLM reasoning during online inference. The performance gains of Cerberus come from two key innovations: motion mask prompting and rule-based deviation detection. The former directs the VLM’s attention to regions relevant to motion, while the latter identifies anomalies as deviations from learned norms rather than enumerating possible anomalies. Extensive evaluations on four datasets show that Cerberus on average achieves 57.68 fps on an NVIDIA L40S GPU, a 151.79 × \times speedup, and 97.2% accuracy comparable to the state-of-the-art VLM-based VAD methods, establishing it as a practical solution for real-time video analytics. † † cc-license: by 1. Introduction Video anomaly detection (VAD) is a cornerstone task in video analytics that identifies unusual activities, such as traffic accidents or violent behaviors, with broad applications in public safety, traffic management, and smart surveillance ( Liu et al. 2025 ; Ramachandra et al. 2020 ; Sultani et al. 2018a ) . The rise of large language models (LLMs) and vision-language models (VLMs) has opened new possibilities for VAD (Figure 1 ). Compared with conventional methods, VLM-based VAD offers two main advantages: • From recognition to open-ended comprehension. Traditional VAD relies on Deep Neural Networks (DNNs) that output data of predefined and fixed categories (e.g., object counts, bounding boxes, action labels). These systems can answer “what is present”, but struggle to connect events into physical contexts. VLMs, by combining visual perception with linguistic knowledge from large-scale pretraining, enable deeper comprehension. They can perform causal inference, retrieve contextual details, generate human-interpretable explanations, and hence provide a more flexible interface and finer granularity for anomaly detection. For example, instead of just detecting “a running person”, a VLM can infer in what situation the person is running and whether it is abnormal. • Flexible anomaly definition via natural language. Traditional video analytics systems require complex and careful query planning, which requires extensive domain-specific experience ( Zhang et al. 2024 ; Bhardwaj et al. 2022 ; Zhang et al. 2017 ; Jiang et al. 2018 ) . For example, detecting “a person chasing another with a weapon” may involve manual pipeline construction, tuning, and cross-platform deployment of modules including motion detector, object detectors, action recognizers, and trackers. On the other hand, VLM-based systems allow users to specify conditions directly in natural language, such as “a person chasing another with a weapon in a crowded street”. It makes configuration simpler and more intuitive, leading to a lowered entry bar and bootstrapping cost. Figure 1 . Two common VLM-based VAD pipelines. Top: a modular two-step design where a VLM describes video content and an LLM reasons ( Zanella et al. 2024 ; Ye et al. 2025 ; Yang et al. 2024a ) . Bottom: an integrated single-step design where a full-fledged VLM like Gemini ( Team et al. 2023 ) and GPT-4o ( Achiam et al. 2023 ) handles both perception and reasoning ( Zhao et al. 2025 ; Li et al. 2024 ) . While VLMs exhibit strong capabilities for VAD, their application presents critical challenges: (1) Prohibitive computational cost. The massive scale of VLM architectures inherently demands substantial computational resources ( Zheng et al. 2024 ) . In addition to the overhead incurred by the enormous size of the network, VLMs introduce extra overhead from cross-modal alignment ( Feng et al. 2025 ; Li et al. 2022c ) and autoregressive decoding ( You et al. 2024 ; Bae et al. 2023 ) , both of which increase latency and memory usage. For example, in our experiments on an NVIDIA L40s GPU ( NVIDIA 2025 ) , processing 10 frames with Qwen2.5-VL-7B ( Bai et al. 2025b ) takes 8.48s and 17.85 GB of memory. This is about 20 times slower and heavier than modern DNN-based detectors like YOLOv10-L ( Wang et al. 2024 ) , which uses only 0.43s and 861 MB. (2) Susceptibility to distraction in multimodal grounding. Although natural language interfaces simplify anomaly specification, VLMs often show unstable grounding in complex scenes. This stems from pretraining, where alignment is shaped by captions that emphasize salient objects (e.g., large, bright, central), causing them to neglect subtle but critical cues ( Xiao et al. 2024 ) . Additionally, spurious correlations and spatial-overlap heuristics can further divert attention to irrelevant regions ( Reich and Schultz 2024 ; Zong et al. 2025 ) . In VAD, this is detrimental: for instance, a small peripheral car running a red light may be ignored if a nearby large bus dominates attention. (3) Lack of design in contextualizing VLMs for anomaly detection. Conventional DNN-based pipelines excel at incorporating scene-specific priors, such as normality learned from background subtraction or trajectory clustering, which can enhance accuracy and adaptability across environments ( Jiang et al. 2021 ; Noghre et al. 2024 ) . However, the design of contextualizing VLM-based VAD methods is still in its infancy. Existing VLM-based VAD solutions rely mainly on models pretrained on general knowledge and prompt engineering ( Zanella et al. 2024 ; Ye et al. 2025 ) . Methods that attempt to improve accuracy by asking LLMs to enumerate possible anomalies ( Yang et al. 2024a ; Ding et al. 2025 ) are also fragile. For example, in a traffic-monitoring scenario, a non-contextualized VLM which can identify “accidents” or “assaults” could easily overlook a contextual anomaly like “skateboarding on a pedestrian-only lane”. To deal with these fundamental challenges and democratize VLMs for anomaly detection, we propose Cerberus , a real-time VLM-based VAD system that combines lightweight perception and deep comprehension. The design of Cerberus builds on three core mechanisms tailored for VLM and VAD: cascaded architecture, motion-mask prompting, and rule-based deviation detection. First, to address prohibitive computational costs, we investigate how to minimize redundant inference. Video streams are highly redundant and applying expensive multimodal alignment and decoding to all segments indiscriminately is extremely inefficient. Inspired by how humans skim ordinary scenes and only focus on unusual ones, we design a cascaded pipeline that filters out redundant frames while keeps key semantic information. Specifically, a lightweight Contrastive Language-Image Pretraining (CLIP) ( Radford et al. 2021 ) model performs coarse filtering to discard irrelevant frames, while a powerful VLM conducts fine-grained reasoning on the remaining candidates. The cascaded pipeline is carefully designed to allow the lightweight stage to trade precision for high recall, and hence anomalous content is preserved while the overhead is significantly reduced. Second, to mitigate distraction in multimodal grounding, we examine how to strengthen focus on relevant regions. VLMs sometimes over-attend to salient but irrelevant elements, missing subtle cues that are decisive for VAD. It presents a critical question: how can we guide the model to focus on relevant regions? We observed

Read the complete text