Sparse Reasoning is Enough: Biological-Inspired Framework for Video Anomaly Detection with Large Pre-trained Models
He Huang, Zixuan Hu, Dongxiao Li, Yao Xiao, Ling-Yu Duan · 2025
Read Sparse Reasoning is Enough: Biological-Inspired Framework for Video Anomaly Detection with Large Pre-trained Models by He Huang, Zixuan Hu, Dongxiao Li, Yao Xiao, Ling-Yu Duan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17094.txt.
Introduction
He Huang and four co-authors question an expensive habit in video anomaly detection: examining every frame. Their system, ReCoVAD, watches surveillance-style video for unusual events but reasons in depth about only a small share of frames. The design copies the human nervous system, which pairs a fast reflex pathway with a slower, conscious one.
A light module first scores frames quickly, consulting a memory of past frames and their scores. Only novel frames go to a vision-language model, which writes event descriptions and refined scores, while a large language model periodically reviews those descriptions to catch new anomaly types and fix mistakes. On two standard datasets the system matched the best training-free results while processing roughly a sixth to a quarter of the frames rivals used. Sparse reasoning, the authors argue, is enough.
Catalog record
| CREATOR | He Huang, Zixuan Hu, Dongxiao Li, Yao Xiao, Ling-Yu Duan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17094
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sparse Reasoning is Enough: Biological-Inspired Framework for Video Anomaly Detection with Large Pre-trained Models He Huang, Zixuan Hu, Dongxiao Li, Yao Xiao, Ling-Yu Duan Abstract Video anomaly detection (VAD) plays a vital role in real-world applications such as security surveillance, autonomous driving, and industrial monitoring. Recent advances in large pre-trained models have opened new opportunities for training-free VAD by leveraging rich prior knowledge and general reasoning capabilities. However, existing studies typically rely on dense frame-level inference, incurring high computational costs and latency. This raises a fundamental question: Is this dense reasoning truly necessary when using powerful pre-trained models in VAD systems? To answer this, we propose ReCoVAD , a novel framework inspired by the human nervous system’s dual re flex and co nscious pathways, enabling selective frame processing to reduce redundant computation. ReCoVAD consists of two core pathways: i) The Reflex pathway, a lightweight CLIP-based component, fuses visual features with prototype prompts to produce decision vectors. These are used to query a dynamic memory of past frames and their anomaly scores, enabling the system to rapidly determine whether to respond immediately or escalate the frame for further reasoning. ii) The Conscious pathway, a medium-scale vision-language model, generates textual event descriptions and refined anomaly scores for novel frames. It continuously updates the memory and prototype prompts, while an integrated LLM periodically reviews accumulated descriptions to identify unseen anomalies, correct errors, and refine prototypes. Our extensive experiments show that ReCoVAD reaches state-of-the-art training-free performance while processing only 28.55% / 16.04% of frames used by the previous methods in UCF-Crime/XD-Violence datasets, demonstrating that sparse reasoning is enough for effective large-model-based VAD. 1 Introduction Video Anomaly Detection (VAD) aims to automatically identify anomalous events in video streams that exhibit deviation from normal patterns. It has attracted substantial attention due to its wide-range applications, including security surveillance 19 ; 34 , autonomous driving 47 ; 13 , and industrial monitoring 26 , etc. Due to limited data and computational resources in the edge devices (e.g., camera, vehicle), it’s crucial to develop efficient and generalizable VAD systems to enable in-time responses to diverse anomalies. Figure 1: (a) The nervous system routes familiar signals through low-cost reflex arcs, while allocating cortical resources only to novel or complex inputs via the thalamus–cortex loop. The Bidirectional feedback enables top-down modulation of reflexes and bottom-up filtering of redundant information (b) ReCoVAD mirrors this architecture: the reflex pathway filters redundant frames by leveraging visual signals and prompt-conditioned decision vectors, while the conscious pathway applies VLM/LLM-based reasoning for novel inputs. Through a feedback loop, it refines both memory and prompts, progressively enhancing selectivity. While conventional VAD models exhibit high efficiency in limited conditions, they suffer from poor generalization and remain fragile in dynamic real-world deployments 54 ; 21 ; 50 ; 43 ; 12 . To overcome this, recent works 50 ; 46 ; 15 ; 14 ; 35 ; 6 explore the integration of Large Visual Language Model (LVLM) and Large Language Model (LLM), leveraging their broad prior knowledge from the large-scale pre-training. This emerging paradigm involves using i) LVLMs to produce rich textual descriptions of video frames and ii) LLMs to infer anomalous events from these descriptions 50 . By decoupling anomaly detection from low-level visual cues, this system achieves significantly improved generalization. Despite these advantages, large-model VAD systems face two critical deployment challenges: i) High computational cost of inference 30 ; 53 , and ii) Dense frame-level analysis to ensure anomaly coverage. These challenges inevitably compound the overall computational burden. This raises a fundamental question: When leveraging powerful foundation models, is frequent inference across densely sampled frames truly necessary? While prior studies have validated the benefits of dense coverage in traditional VAD pipelines 50 ; 46 ; 43 , its utility in large-model systems remains underexplored and potentially suboptimal. To address this challenge, we draw inspiration from how the human nervous system reduces the brain’s burden through hierarchical processing 24 , allocating cognitive resources selectively. As shown in Figure 1 (a), it employs two complementary processing pathways: i) Reflex pathway that rapidly filters and responds to familiar, redundant signals via reflex arcs and ii) Conscious pathway where the thalamus performs primary analysis 5 and cortical regions engage in further reasoning only on novel or complex inputs 25 . These pathways interact through a bidirectional feedback loop 48 : the conscious pathway influences reflex responses through memory and knowledge, while the reflex filters out redundant signals, reducing the cognitive load for the conscious pathway. This paradigm suggests that dense and uniform reasoning is not always necessary, inspiring us to develop a more efficient, selective information routing framework for VAD. To this end, we propose ReCoVAD , a dual-pathway framework that enhances model collaboration to construct the loop of efficient “ Re flex” filtering with deeper “ Co nscious” reasoning in Figure 1 (b). The reflex pathway leverages a lightweight CLIP 28 model to fuse visual features with prompts derived from textual event prototypes, producing a decision vector to query a dynamic memory of representative frames’ records (decision vectors- detection scores). If a frame falls within the coverage of the memory, it is regarded as redundant, and its anomaly score is retrieved directly, emulating low-latency, reflex filtering. Frames that fail this check are escalated to the conscious pathway, where a medium-scale (7B) VLM generates textual event descriptions and anomaly scores under the guidance of the textual event prototypes. These outputs are then written back into the reflex pathway’s memory, progressively refining the system’s internal representations. To complete the feedback loop, the conscious pathway also includes an LLM-based reasoner that periodically revisits the generated descriptions to identify novel anomalies, revise earlier decisions, and adapt the prototype prompts, thus realizing a top-down refinement coupled with bottom-up filtering. Therefore, our contributions are summarized as follows: (1) To our best knowledge, we are the first to propose a biological-inspired VAD framework that simulates the reflex–conscious pathways of the human nervous system, significantly reducing the computational burden of using VLM/LLM while maintaining detection performance. (2) In this framework, we establish a novel closed-loop architecture that couples bottom-up filtering with top-down refinement: a lightweight reflex path rapidly filters redundant frames via memory querying, while a VLM-LLM conscious path processes novel frames and dynamically refines memory and prompts to enable a self-evolving VAD system. (3) Through our extensive experiments, we demonstrate that dense-frame inference is unnecessary for large-model-based VAD. Our ReCoVAD achieves state-of-the-art, training-free performance on the UCF-Crime and XD-Violence datasets, while using only 28.55% and 16.04% of the frames compared to previous large-model-based approaches. 2 Related Work 2.1 Video Anomaly Detection Video Anomaly Detection (VAD) methods are commonly grouped into one-class, unsupervised, and weakly supervised settings, based on the availability of anomalies and labels during training. One-class VAD assumes access to only normal videos and models normal patterns via direct statistical modeling 2 ; 4 ; 10 or implicitly through proxy tasks 10 ; 19 ; 44 ; 32 ; 33 ; 20 ; 27 ; 16 ; 36 ; 7 , including future frame prediction 19 ; 44 , reconstruction 32 ; 27 , and contrastive learning 16 ; 36 . Unsupervised VAD, on the other hand, assumes a mixture of normal and abnormal videos without