Dual Conditioned Motion Diffusion for Pose-Based Video Anomaly Detection
Hongsong Wang, Andi Xu, Pinle Ding, Jie Gui · 2024
Read Dual Conditioned Motion Diffusion for Pose-Based Video Anomaly Detection by Hongsong Wang, Andi Xu, Pinle Ding, Jie Gui (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17210.txt.
Introduction
Hongsong Wang and three co-authors present a 2024 paper on spotting odd behavior in video by watching body poses. Rebuilding what it sees helps a system flag strange patterns, while predicting ahead helps it flag strange trends. Prior pose-based detectors usually picked one route. Their framework, DCMD, takes both: a diffusion model conditioned on the observed motion and on a learned summary of it.
A motion transformer inside the diffusion process reads layered features of the movement. A custom penalty sharpens the gap between normal and abnormal samples, using both time-wise and global links. At test time, a mask completion step strengthens the prediction side. On four data sets the method beat state-of-the-art rivals by clear margins and generalized well. The authors target surveillance and safety monitoring that must respect privacy by using poses, not faces.
Catalog record
| CREATOR | Hongsong Wang, Andi Xu, Pinle Ding, Jie Gui |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17210
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dual Conditioned Motion Diffusion for Pose-Based Video Anomaly Detection Hongsong Wang, Andi Xu, Pinle Ding, Jie Gui Abstract Video Anomaly Detection (VAD) is essential for computer vision research. Existing VAD methods utilize either reconstruction-based or prediction-based frameworks. The former excels at detecting irregular patterns or structures, whereas the latter is capable of spotting abnormal deviations or trends. We address pose-based video anomaly detection and introduce a novel framework called Dual Conditioned Motion Diffusion (DCMD), which enjoys the advantages of both approaches. The DCMD integrates conditioned motion and conditioned embedding to comprehensively utilize the pose characteristics and latent semantics of observed movements, respectively. In the reverse diffusion process, a motion transformer is proposed to capture potential correlations from multi-layered characteristics within the spectrum space of human motion. To enhance the discriminability between normal and abnormal instances, we design a novel United Association Discrepancy (UAD) regularization that primarily relies on a Gaussian kernel-based time association and a self-attention-based global association. Finally, a mask completion strategy is introduced during the inference stage of the reverse diffusion process to enhance the utilization of conditioned motion for the prediction branch of anomaly detection. Extensive experiments on four datasets demonstrate that our method dramatically outperforms state-of-the-art methods and exhibits superior generalization performance. Code — https://github.com/guijiejie/DCMD-main Introduction Figure 1: Illustration of video anomaly detection with normal instances (green) and abnormal instances (red). Video Anomaly Detection (VAD) is an essential topic in computer vision and security applications. Different from human action understanding ( Wang and Wang 2018 ; Chen, Wang, and Gui 2023 ; Wang, Zhao, and Gui 2024 ; Weng et al. 2025 ) , VAD enables prompt identification of abnormal occurrences, such as human actions, accidents, and illnesses. Anomalies are generally characterized as uncommon, unexpected, or unusual phenomena that exhibit significant deviations from normality. Conversely, normality is defined as what is expected and regularly encountered. Detecting video anomalies can be challenging as these events occur infrequently and belong to various categories. Labeling data is costly and time-consuming, making collecting all possible abnormal samples impractical for fully supervised learning approaches. Consequently, anomaly detection problems are typically treated as One Class Classification (OCC) ( Zaheer et al. 2020 ) . Only normal data is utilized during model training, and data that deviates significantly from the normal pattern is identified as an anomaly during testing ( Cai et al. 2021 ) . There has been a growing trend toward detecting abnormal events or behaviors by analyzing human poses extracted from video frames ( Flaborea et al. 2023 ) , as illustrated in Figure 1 . Utilizing skeletons to depict human motions in videos is a highly effective approach to protecting privacy and circumventing the limitations of appearance-based attributes. Besides privacy protection, skeletal features are compact, well-structured, and highly descriptive of human motion ( Morais et al. 2019 ) . Existing VAD works are mainly categorized into reconstruction-based and prediction-based methods. Reconstruction-based methods ( Li, Chang, and Liu 2021 ; Chang et al. 2020 ) first compress the input data into a low-dimensional representation and then recover the original data from this representation. By comparing the reconstruction error between the original data and the recovered data, anomalies such as irregular patterns or structures in the data can be detected. Prediction-based methods ( Liu et al. 2017 ) focus on predicting future frames or events based on history data, and anomalies are indicated when future data deviates from the predicted trends. These methods are effective in modeling and uncovering the temporal connections between successive frames. Nevertheless, they may be prone to noise ( Tang et al. 2020 ) . Recently, diffusion models have shown great success in generating high-quality samples ( Ho, Jain, and Abbeel 2020 ) . One of the outstanding capabilities of diffusion models is the flexibility to handle a wide range of input conditions that can encompass different types of data, enabling customized generation tasks ( Ho and Salimans 2021 ) . Due to the inherent multimodality of human motion and the diversity of both normal and abnormal patterns, diffusion models are naturally well-suited for modeling human motion and detecting anomalies. However, simply applying diffusion models for VAD suffers inherent limitations of reconstruction-based or prediction-based approaches. To address the above issues, we present a unified framework that seamlessly combines the advantages of both reconstruction-based and prediction-based approaches for pose-based VAD. The reconstruction is based on an auto-encoder architecture, whereas the prediction utilizes a diffusion model. We propose a novel Dual Conditioned Motion Diffusion (DCMD) that seamlessly integrates conditioned motion and conditioned embedding, enabling the exploitation of both pose characteristics and latent semantics of observed movements. During the reverse diffusion process, we introduce a motion transformer specifically designed to extract potential correlations from multi-layered spectrum features of human motion. In addition to reconstruction and prediction losses, we devise a United Association Discrepancy (UAD) regularization that leverages Gaussian kernel-based time association and self-attention-based global association. Furthermore, during the inference stage, we employ a mask completion strategy to bolster the utilization of conditioned motion for prediction-based anomaly detection. Experiments conducted on popular human-related anomaly detection datasets demonstrate the superior performance of our proposed method compared to state-of-the-art approaches. Our main contributions are summarised below: • We introduce a novel framework that seamlessly integrates reconstruction-based and prediction-based methods for video anomaly detection, leveraging the strengths of both approaches. • We propose a Dual Conditioned Motion Diffusion (DCMD), which incorporates both conditioned motion and conditioned embedding in a diffusion-based model. • We propose a motion transformer for anomaly detection with a novel regularization that uses both time association and global association to improve the discriminability between normal and abnormal instances. • We present a mask completion method during the denoising process of diffusion, enabling more effective utilization of observed motions while predicting future motions for anomaly detection. Related Work We summarize previous work on Video Anomaly Detection (VAD) from three aspects: reconstruction-based, prediction-based, and pose-based. Reconstruction-Based Methods: Reconstruction-based methods comprise two components: an encoder and a decoder. The encoder compresses the input frame into a low-dimensional feature while the decoder reconstructs the output from this compressed representation. Reconstruction error is a criterion for distinguishing between normal and abnormal events. Luo et al. ( Luo et al. 2019 ) introduce a novel deep neural network architecture that leverages sparse coding, incorporates a temporal coherence term to preserve similarity between similar frames, and employs a stacked recurrent neural network to optimize sparse coefficients for achieving real-time anomaly detection. Li et al. ( Li, Chang, and Liu 2021 ) present a two-stream network designed to capture the visual and motion characteristics of typical events in videos. To encode the scene, objects as well as motion information, Chang et al. ( Chang et al. 2020 ) propose a novel deep k-mean clustered convolutional self-encoder architecture. There are also some methods utilizing architectures other than convolutional neural networks. Gong et al. ( Gong et al. 2019 ) introduce a memory-enhanced autoencoder that can distinguish new test samples by memorizing prototypical normal data elements during training. Doshi et al. ( Doshi and Yılmaz 2022 ) propose a dual-stage approach combining deep learning with a kNN-based RNN to overcome forgetting in end-to-end models, enabling efficient continual learning. Zhong et al.