RobustA: Robust A nomaly Detection in Multimodal Data
Salem AlMarri, Muhammad Irzam Liaqat, Muhammad Zaigham Zaheer, Shah Nawaz, Karthik Nandakumar, Markus Schedl · 2025
Read RobustA: Robust A nomaly Detection in Multimodal Data by Salem AlMarri, Muhammad Irzam Liaqat, Muhammad Zaigham Zaheer, Shah Nawaz, Karthik Nandakumar, Markus Schedl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07276.txt.
Introduction
Salem AlMarri and five co-authors confront a useful weakness of multimodal anomaly detection: the real world's inputs arrive corrupted. Systems that fuse video and audio beat video-only models in the lab, but environmental distortions can damage one or both modalities after deployment, and nobody had systematically measured the damage. The team built RobustA, a curated evaluation dataset that applies controlled audio and visual corruptions so the effects on detection can be observed.
They also propose a detection method built for this reality. It learns a shared representation space across modalities and, at inference time, dynamically weights each modality according to its estimated corruption level, leaning on whichever channel is cleaner. The dataset, with its extracted features, will be public, and the method showed notable resilience against corrupted inputs.
Catalog record
| CREATOR | Salem AlMarri, Muhammad Irzam Liaqat, Muhammad Zaigham Zaheer, Shah Nawaz, Karthik Nandakumar, Markus Schedl |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07276
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RobustA: Robust A nomaly Detection in Multimodal Data Salem AlMarri, Muhammad Irzam Liaqat, Muhammad Zaigham Zaheer, Shah Nawaz, Karthik Nandakumar, Markus Schedl Abstract † † † \dagger Equal contribution In recent years, multimodal anomaly detection methods have demonstrated remarkable performance improvements over video-only models. However, real-world multimodal data is often corrupted due to unforeseen environmental distortions. In this paper, we present the first-of-its-kind work that comprehensively investigates the adverse effects of corrupted modalities on multimodal anomaly detection task. To streamline this work, we propose RobustA , a carefully curated evaluation dataset to systematically observe the impacts of audio and visual corruptions on the overall effectiveness of anomaly detection systems. Furthermore, we propose a multimodal anomaly detection method, which shows notable resilience against corrupted modalities. The proposed method learns a shared representation space for different modalities and employs a dynamic weighting scheme during inference based on the estimated level of corruption. Our work represents a significant step forward in enabling the real-world application of multimodal anomaly detection, addressing situations where the likely events of modality corruptions occur. The proposed evaluation dataset with corrupted modalities and respective extracted features will be made publicly available. I Introduction Video anomaly detection (VAD) is a challenging computer vision task with various real-world applications including surveillance, autonomous navigation, packaging, and biomedical imaging [ 18 ] . Generally, VAD methods aim to predict high anomaly scores for the frames in a video that deviate significantly from the norm, where the application context determines the norm. For example, events such as shoplifting or violence can be considered anomalies in the CCTV surveillance context [ 28 ] . Since it is laborious to obtain fine-grained (pixel-level or frame-level) labels of anomalies in videos, a weakly-supervised VAD (WS-VAD) setting that learns to detect anomalous frames using only video-level binary labels is typically used. In WS-VAD, a training video is labeled as normal if no anomalous event is present, whereas it is labeled as an anomaly if any anomalous event is present [ 26 , 4 , 35 ] . In recent years, WS-VAD has gained considerable popularity and became a mainstream VAD paradigm [ 25 , 24 , 26 ] . More recently, WS-VAD has been transformed into a multimodal learning task by including other informative modalities such as audio to discriminate and locate events better [ 28 , 33 ] . For instance, it is challenging to rely only on visual signals in shaky videos covering an explosion event. However, if accompanied by the audio modality, the event can be easily classified. Prior works have successfully shown the effectiveness of using video and audio pairs to improve the overall performance of WS-VAD [ 28 , 29 , 7 ] . Fig. 1: Existing multimodal anomaly detection approaches [ 28 , 38 , 27 ] are not robust when subjected to corruptions in either modality, including vision (e.g., fog and motion blur) or audio (e.g., babble). Despite their many advantages, deep neural network models are vulnerable to typical corruptions in the input data (e.g., degradations, distortions, and disturbances caused by weather changes, system errors, etc.) [ 11 ] . Though multimodal learning systems achieve better performance on multimodal data [ 30 ] compared to their unimodal counterparts, they are also susceptible to such vulnerabilities. Multimodal VAD systems, in particular, are prone to such issues (see Fig. 1 ), as they are often subjected to extreme environmental conditions [ 32 ] . For this reason, it is essential to design methods that can withstand corrupted data during deployment. While it may be possible to achieve this goal by including corrupted data during the training process, this work aims to evaluate the robustness of multimodal VAD systems trained only on clean data, when encountering corrupted data during deployment . Existing literature lacks a detailed study and appropriate dataset that investigates the impact of corrupted modalities on WS-VAD and multimodal VAD systems. In this work, we empirically investigate this problem with a proposed dataset, RobustA , specifically to study this issue. We observe that existing multimodal anomaly detection models deteriorate dramatically under corrupted data. For example, in Fig. 1 , the performance of XDVioDet [ 28 ] drops from 78.36 % to 58.70 % when exposed to fog in visual modality during testing. To alleviate this issue, we propose a robust anomaly detection method that is resilient against corrupted modalities. The proposed method independently maps multiple modalities into a shared representation space. This shared space can be particularly beneficial in scenarios where one modality is compromised. For example, if one modality becomes distorted, inaccurate, noisy, or is entirely missing, the shared space can leverage information from the remaining modalities to compensate for the loss. Results reported in Section V-B suggest that our approach not only yields better multimodal performance but also demonstrates resilience against corruptions. The key contributions of our work are as follows: • We present RobustA , a first-of-its-kind, carefully curated dataset comprising 8 visual corruptions and 8 audio corruptions, enabling evaluation of the multimodal anomaly detection methods against various types and levels of modality corruptions. • We propose an anomaly detection approach that independently maps multiple modalities to learn representations in a shared space while dynamically adjusting the weights of individual modalities to reduce the impact of the compromised modality towards multimodal anomaly detection. • We study the plug-and-play nature of our proposed approach, demonstrating its resilience against compromised modalities when used in conjunction with existing approaches. II Related Work II-A Weakly Supervised Video Anomaly Detection and Multimodality Weakly Supervised Video Anomaly Detection (WS-VAD) refers to the process of identifying abnormal events in a given video while the training is carried out using only video-level binary labels. The problem was first introduced by Sultani et al. [ 26 ] where the authors utilized multi-instance learning based ranking to carry out the overall training. The research was advanced by several researchers [ 36 , 37 , 34 ] . However, most of these studies are focused on unimodal (video-only) training and testing. More recently, anomaly detection task is transformed into multimodal learning where more than one modalities (usually vision and audio) are present during training and testing. For example, XDVioDet [ 28 ] utilized a multi-branch neural network comprising holistic, localized, and score branches to effectively leverage multimodal feature representations. Similarly, HyperVD [ 38 ] proposed a method using two graph-based hyperbolic convolutional networks for spatio-temporal feature extraction from multimodal input. While the multimodal anomaly detection task has gained popularity due to superior performance compared to the unimodal counterpart approaches, existing literature lacks rigorous studies understanding the adverse effects of compromised data on multimodal learning systems. As, CCTV surveillance cameras equipped with anomaly detection systems are highly prone to environmental corruptions, it is pertinent to explore this novel research direction. II-B Addressing Compromised Modalities Corrupted Modalities. Data collected from diverse multiple sources may contain corrupted samples due to various factors, such as sensor failure, obstructions in the video stream, noise in audio signals, data storage errors, and many more. Recent years have seen an increased interest in investigating the vulnerability of deep models against modality corruptions [ 23 , 3 , 13 , 31 , 15 ] . For example, Hendrycks