Flashback: Memory-Driven Zero-shot, Real-time Video Anomaly Detection
Hyogun Lee, Haksub Kim, Ig-Jae Kim, Yonghun Choi · 2025
Read Flashback: Memory-Driven Zero-shot, Real-time Video Anomaly Detection by Hyogun Lee, Haksub Kim, Ig-Jae Kim, Yonghun Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15205.txt.
Introduction
Watching hours of camera feed for the rare odd event is slow work, and trained systems often fail when the scene changes. Hyogun Lee and co-authors present Flashback in a 2025 paper, a method that needs no training on crime or violence clips. It works in two stages that copy how a person judges a scene from past experience.
First, a large language model writes short captions of normal and odd scenes, building a memory bank offline. Then each new clip is turned into numbers and matched against that bank by closeness. No language model runs during live use, so the system stays fast even on a home-grade graphics card. On two large real test sets it beat prior zero-shot methods by wide margins, a strong sign for live use.
Catalog record
| CREATOR | Hyogun Lee, Haksub Kim, Ig-Jae Kim, Yonghun Choi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15205
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Flashback: Memory-Driven Zero-shot, Real-time Video Anomaly Detection Hyogun Lee, Haksub Kim, Ig-Jae Kim, Yonghun Choi Abstract Video Anomaly Detection (VAD) automatically identifies anomalous events from video, mitigating the need for human operators in large-scale surveillance deployments. However, two fundamental obstacles hinder real-world adoption: domain dependency and real-time constraints—requiring near-instantaneous processing of incoming video. To this end, we propose Flashback, a zero-shot and real-time video anomaly detection paradigm. Inspired by the human cognitive mechanism of instantly judging anomalies and reasoning in current scenes based on past experience, Flashback operates in two stages: Recall and Respond. In the offline recall stage, an off-the-shelf LLM builds a pseudo-scene memory of both normal and anomalous captions without any reliance on real anomaly data. In the online respond stage, incoming video segments are embedded and matched against this memory via similarity search. By eliminating all LLM calls at inference time, Flashback delivers real-time VAD even on a consumer-grade GPU. On two large datasets from real-world surveillance scenarios, UCF-Crime and XD-Violence, we achieve 87.3 AUC (+7.0 pp) and 75.1 AP (+13.1 pp), respectively, outperforming prior zero-shot VAD methods by large margins. 1 Introduction Video Anomaly Detection (VAD) automatically identifies events that deviate from learned normal patterns in continuous video streams, overcoming the impracticality of manual monitoring in public safety [ 63 ] , intelligent transportation [ 5 ] , and industrial inspection systems [ 37 ] . Currently, the number of surveillance cameras deployed worldwide is growing rapidly [ 8 , 18 , 39 ] , generating volumes of video data that far exceed human monitoring capacity and making timely anomaly detection all but impossible. Therefore, the practical adoption of automated VAD is urgently needed. Real-world VAD deployment faces two fundamental obstacles: domain dependency and real-time constraints. First, nearly all VAD paradigms—whether weakly-supervised [ 39 , 58 ] , one-class [ 19 , 30 ] , or unsupervised [ 42 , 46 ] —require collecting and annotating domain-specific footage followed by model retraining for each new environment, imposing prohibitive time and cost burdens when target-domain samples are unavailable. Second, since emergencies can occur at any time, processing each video segment should finish before the next one arrives. Otherwise, in real-world scenarios with continuously arriving segments, some segments would have to be skipped to avoid latency accumulation, undermining uninterrupted anomaly detection. Figure 1 : Bridging speed and reasoning. (a) Real-time VAD keeps a light video encoder online but cannot work zero-shot or explain its decisions. (b) Explainable VAD adds a large VLM + LLM in the loop; reasoning is possible, yet speed drops and zero-shot ability is partial. (c) Flashback moves the LLM offline, builds a pseudo-scene memory once, and uses a frozen cross-modal encoder at test time, so it is simultaneously real-time, zero-shot, and explainable. (d) On XD-Violence [ 54 ] (XD), this design lifts AP by 13 percentage point and boosts throughput 34 \times over the prior state-of-the-art. Recent research has addressed the challenges of domain generalization and real-time processing separately. Zero-shot VAD techniques leverage pre-trained vision-language models [ 29 ] (VLMs) and large language models [ 45 ] (LLMs) to avoid any domain-specific retraining. Among these, caption-and-score methods [ 16 , 60 ] first generate segment captions via a VLM and then compute anomaly scores with an LLM, but they suffer from the heavy computation of autoregressive captioning and noisy text outputs. Prompt-based approaches [ 1 , 57 , 58 ] reduce LLM invocations by injecting optimized text prompts into the VLM inference stage, improving efficiency and domain transfer. However, they often struggle to maintain coherent anomaly scores across temporally contiguous segments and remain sensitive to prompt vocabulary design. Parallel efforts in real-time VAD aim to process each fixed-length segment before the next one arrives. End-to-end weakly supervised models [ 23 ] accelerate inference but still fall short of sub-second segment processing for unseen domains, while density-estimation detectors [ 32 ] achieve per-segment delays of around 200 ms yet require domain-specific model updates to maintain accuracy. Although zero-shot methods provide domain-agnostic adaptability and real-time ones deliver low latency, no existing approach unifies both capabilities. In this paper, we propose the Flashback paradigm to improve generalization and ensure consistently low-latency inference. As illustrated in Figure 1 , Flashback comprises two stages: offline pseudo-scene memory construction and online caption-retrieval inference. In the offline stage, Flashback leverages a frozen LLM [ 33 ] to generate captions for a broad spectrum of normal and anomalous scenes without any video input, then converts each caption into an embedding via the video-text cross-modal encoder [ 6 , 15 ] and stores both captions and their embeddings in the memory. In the online stage, incoming video is partitioned into fixed-length segments, whose embeddings are matched against the memory via similarity search to yield segment-level anomaly scores; these scores are then aggregated across segments and smoothed into frame-level predictions. Despite this streamlined pipeline, we face two key challenges. First, the encoder systematically biases representations toward anomalous captions even when processing normal content. To address this, we introduce repulsive prompting : wrapping normal and anomalous captions in distinct prompt templates so that their embeddings remain well separated. Second, a residual skew toward anomalies can persist at inference time. To correct this, we employ scaled anomaly penalization , which attenuates anomalous-caption embedding magnitudes. By fully decoupling the LLM from the online loop, Flashback requires no additional data collection or fine-tuning. Moreover, because anomaly detection finishes each fixed-length segment before the next one arrives—even when this interval is only one second—Flashback enables genuine real-time VAD while providing human-readable explanations via the retrieved captions. We evaluate Flashback on two benchmark datasets—UCF-Crime [ 39 ] and XD-Violence [ 54 ] —and empirically demonstrate that it outperforms zero-shot, unsupervised, and one-class baselines. The contributions of this work are as follows: A novel explainable zero-shot and real-time VAD paradigm . We combine offline pseudo-scene memory construction using a frozen LLM with online caption retrieval inference to guarantee per-segment processing within its duration without any LLM calls at inference time. Novel techniques for mitigating anomaly bias . We introduce repulsive prompting , which applies distinct templates to normal and anomalous captions to prevent embedding collapse and enhance representation separation, and scaled anomaly penalization , which attenuates anomalous-caption embedding magnitudes at inference time. Zero-shot pseudo-scene memory without real anomaly data . Our pseudo-scene memory is built solely from LLM-generated captions—without using any actual anomalous video footage—yet still spans a broad range of normal and anomalous scenes. SOTA performance at real-time speed . Even for segments as short as one second, our anomaly detector processes each in under one second—enabling genuine real-time VAD—and outperforms one-class and unsupervised methods on the UCF-Crime and XD-Violence. 2 Related work Video anomaly detection. Early video anomaly detection (VAD) methods minimize reconstruction or other generative losses on normal-only or unlabeled footage [ 19 , 30 , 42 , 46 , 47 , 48 , 59 ] and thus treat large reconstruction error as a signal of abnormality. A parallel line formulates VAD as weakly-supervised multiple-instance learning, using video-level labels but no reliable frame labels [ 39 , 62 , 53 ] . To broaden coverage, later work incorporates audio cues [ 54 , 56 ] or instruction-tunes