Learning Where and When: Patch-Based Spatiotemporal Localization in Weakly Supervised Video Anomaly Detection

Hamza Karim, Nghia Nguyen, Lokman Bekit, Yasin Yilmaz · 2026

Read Learning Where and When: Patch-Based Spatiotemporal Localization in Weakly Supervised Video Anomaly Detection by Hamza Karim, Nghia Nguyen, Lokman Bekit, Yasin Yilmaz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29498.txt.

Introduction

Karim, Nguyen, Bekit, and Yilmaz extend weakly supervised video anomaly detection, which learns from videos labelled only as normal or abnormal overall. Most work finds when an anomaly happens but ignores where it sits inside the frame. That spatial gap hurts real deployment, where operators need to see the exact region involved.

Their patch-based framework scores small grid regions of each frame under multiple instance learning, which trains from bag-level labels alone. A strategy called Proximity-Aware Top-k selection picks the most suspicious regions while respecting their closeness in space and time. The model thus produces fine-grained spatial anomaly maps without any bounding-box supervision during training. The authors report that it surpasses existing state-of-the-art methods across several benchmarks, with large gains. They also release test-set box annotations, code, and pretrained models.

Catalog record

CREATORHamza Karim, Nghia Nguyen, Lokman Bekit, Yasin Yilmaz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29498

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Where and When: Patch-Based Spatiotemporal Localization in Weakly Supervised Video Anomaly Detection Hamza Karim, Nghia Nguyen, Lokman Bekit, Yasin Yilmaz Abstract Weakly supervised video anomaly detection (WSVAD) has predominantly focused on temporal localization, identifying when anomalies occur while largely neglecting their spatial extent within frames. Yet, spatial localization is essential for interpretability and practical deployment in real-world settings. We introduce a patch-based spatiotemporal framework for weakly supervised anomaly localization that jointly models where and when anomalies occur. Our approach operates on grid-level patch features and learns region-level anomaly scores under a multiple instance learning paradigm. We further propose a Proximity-Aware Top-k spatiotemporal selection strategy that enables the model to generate fine-grained spatial anomaly maps without requiring bounding-box supervision during training. Our method surpasses existing state-of-the-art approaches across multiple benchmarks, yielding substantial gains in spatiotemporal localization accuracy. In addition, we release frame-level bounding-box annotations for the test sets of two widely used datasets, along with our code and pretrained models, providing new resources to facilitate future research in spatially grounded WSVAD. 1 Introduction Video anomaly detection aims to automatically identify events that deviate from expected patterns in long, untrimmed videos. It plays a central role in applications such as surveillance, public safety, traffic monitoring, and industrial inspection, where rapid and reliable identification of abnormal events is essential. While early approaches primarily focused on frame-level, coarse temporal localization, real-world deployments increasingly require spatiotemporal understanding, i.e., determining not only when an anomaly occurs but also where it manifests within each frame. Obtaining fine-grained spatial annotations for anomalous events is expensive, time-consuming, and often subjective, especially in large-scale datasets. As a result, weakly supervised video anomaly detection (WSVAD), where models are trained using only coarse labels such as video-level annotations, has gained significant attention. The key challenge in this setting is to learn precise spatial localization signals despite the absence of explicit spatial or temporal supervision. Most existing weakly supervised methods primarily focus on temporal detection and rely on global representations, which limits their ability to capture localized abnormal patterns [ 11 ] , [ 4 ] , [ 16 ] , [ 12 ] , [ 3 ] . Detecting anomaly regions within each frame is crucial for several reasons. First, spatial localization greatly improves interpretability, enabling human operators to quickly verify alerts and understand the underlying cause of an anomaly. Second, it supports fine-grained decision making, such as tracking suspicious objects, triggering region-specific responses, or prioritizing computational resources in downstream systems. Third, spatial modeling provides richer contextual cues that help distinguish true anomalies from background clutter, reducing false positives that arise from back-ground biases. In this work, we advance weakly supervised spatiotemporal video anomaly detection through the following contributions: • We propose a novel, patch-based weakly supervised spatiotemporal architecture that produces anomaly scores for patches within each frame , enabling fine-grained spatial localization without requiring bounding box supervision during training. • We introduce a new neighbor-aware top- k MIL loss , where the selected instances are formed through clustering based on temporal and spatial proximity, rather than conventional top- k selection purely based on highest scores. • We provide the bounding box annotations for the widely used anomaly detection datasets XD-Violence and UBnormal [ 14 , 1 ] , enabling standardized evaluation of spatial localization performance in the weakly supervised setting. 2 Related Work 2.1 Weakly Supervised Video Anomaly Detection Multiple instance learning (MIL) has become the dominant paradigm for weakly supervised video anomaly detection, where only video-level labels are available during training. The seminal work of Deep-MIL [ 11 ] formulates anomaly detection as a ranking problem over temporal snippets, training a deep network to assign higher scores to anomalous bags than to normal ones. This framework established the core assumption that anomalous snippets exist within positive bags, and normal snippets dominate negative bags, and has since been extended in numerous directions. Subsequent work addresses key limitations of the original ranking formulation. RTFM [ 12 ] improves discrimination by learning feature magnitudes that better separate anomalous instances, while incorporating temporal context through dilated convolutions and attention mechanisms. URDMU [ 23 ] introduces dual memory units to model the distribution of normal and abnormal features more explicitly. DSANet [ 18 ] further addresses this by disentangling normal and abnormal features at coarse and fine-grained levels. TCN-MIL [ 10 ] extends temporal modeling with non-local operations. CLIPTSA and VADCLIP [ 3 , 16 ] leverage pretrained vision-language representations to align visual features with semantic descriptions, improving generalization to unseen categories. Recently, VERA [ 17 ] adapts frozen VLMs for explainable anomaly detection without fine-tuning. Despite their strong temporal detection performance, these methods uniformly operate at the clip or snippet level, producing a single anomaly score per temporal unit. They offer no mechanism for identifying where within a frame an anomaly occurs, limiting their interpretability and practical utility in surveillance or safety-critical applications. Our work addresses this gap directly by reformulating MIL over spatial patches rather than temporal snippets, enabling joint spatiotemporal scoring under the same weak supervision. 2.2 Spatiotemporal Anomaly Localization Spatial localization under weak supervision has received comparatively little attention, largely due to the absence of bounding-box annotations in standard benchmarks and the ambiguity of anomalous regions. Unsupervised reconstruction-based methods [ 2 , 20 ] implicitly produce spatial error maps by measuring per-pixel or per-patch reconstruction residuals, but these signals are not grounded in semantic anomaly definitions and tend to be noisy in complex scenes. Attention-based unsupervised approaches [ 22 ] identify candidate regions using multi-scale prediction error maps, but rely on heuristic selection rather than learned region-level discrimination. A foundational step toward addressing the weakness in spatial localization was taken by Liu et al. [ 7 ] , who provided bounding-box annotations for the UCF-Crime dataset [ 11 ] and introduced Temporal Intersection over Union (TIoU) as a standardized metric for evaluating spatiotemporal localization. Critically, their analysis also revealed that detectors frequently exploit background contextual cues rather than attending to true anomalous regions, a bias that persists in many subsequent methods and motivates architectures that explicitly model spatial evidence. Building on this foundation, WSSTAD [ 13 ] proposes a dual-branch architecture for weakly supervised spatiotemporal tube localization, using only video-level labels. Through proposal reasoning and progressive refinement, it demonstrates that meaningful spatial grounding is achievable without bounding-box supervision during training. More recently, STPrompt [ 15 ] leverages vision-language prompt embeddings to align local spatial features with semantic anomaly descriptions, combining spatial and temporal streams to further reduce the background bias identified by [ 7 ] . However, existing spatially aware methods share two fundamental limitations. First, they do not model fine-grained grid-level patch features under a MIL objective, which prevents the model from learning region-level anomaly scores in a principled, end-to-end manner. Second, their evaluation is limited due to the lack of spatial annotations in most benchmark datasets; currently, spatial annotations are available only for UCF-Crime and ShanghaiTech [ 19 ] . To address this we propose a patch-based MIL architecture with a consecutive top- k spatiotemporal selection strategy that produces explicit spatial anomaly maps. Furthermore, we provide new spatial annotations for XD-Violence and UBnormal, enabling a more comprehensive and standardized evaluation of spatial localization performance. 3 Methodology Figure 1

Read the complete text