Multimodal Anomaly Detection with a Mixture-of-Experts

Christoph Willibald, Daniel Sliwowski, Dongheui Lee · 2025

Read Multimodal Anomaly Detection with a Mixture-of-Experts by Christoph Willibald, Daniel Sliwowski, Dongheui Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-19077.txt.

Introduction

Christoph Willibald and co-authors build a mixed-expert system that spots faults in robots from several senses at once, in work from 2025. Robot failures come from two roots: the robot's own limits and sudden changes around it. Past detectors watched only one root at a time.

Their system pairs a vision and language watcher for the scene with a motion and force tracker for the robot. A confidence rule picks which expert to trust in each moment. Tests covered home and factory tasks with real robots. The blend caught faults that single-sense systems missed. Such watchers matter as robots spread into homes, warehouses, and care work.

Catalog record

CREATORChristoph Willibald, Daniel Sliwowski, Dongheui Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-19077

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Anomaly Detection with a Mixture-of-Experts Christoph Willibald, Daniel Sliwowski, Dongheui Lee Abstract With a growing number of robots being deployed across diverse applications, robust multimodal anomaly detection becomes increasingly important. In robotic manipulation, failures typically arise from (1) robot-driven anomalies due to an insufficient task model or hardware limitations, and (2) environment-driven anomalies caused by dynamic environmental changes or external interferences. Conventional anomaly detection methods focus either on the first by low-level statistical modeling of proprioceptive signals or the second by deep learning-based visual environment observation, each with different computational and training data requirements. To effectively capture anomalies from both sources, we propose a mixture-of-experts framework that integrates the complementary detection mechanisms with a visual-language model for environment monitoring and a Gaussian-mixture regression-based detector for tracking deviations in interaction forces and robot motions. We introduce a confidence-based fusion mechanism that dynamically selects the most reliable detector for each situation. We evaluate our approach on both household and industrial tasks using two robotic systems, demonstrating a 60% reduction in detection delay while improving frame-wise anomaly detection performance compared to individual detectors. I Introduction As collaborative robots act increasingly autonomously across various applications, accurately monitoring task progress and success becomes crucial. In both household and industrial settings, like the ones depicted in Fig. 1 , autonomous robots are confronted with a range of unknown situations, leading to diverse sources of task failure. Common anomalies arise from hardware defects, human interference, or inaccuracies in the task model. To account for those diverse failure cases, an anomaly detection mechanism must integrate multimodal sensor data to decide whether the current situation constitutes an anomaly. Fig. 1 : Our Mixture of Experts (MoE) framework that combines GMR-based outlier detection with VLM-based scene monitoring to identify anomalies in both household and industrial settings. The framework dynamically selects the most reliable detector for each situation based on a prediction confidence score. Conventional anomaly detectors in robotics, whether online or offline, typically follow two approaches: modeling the statistical distribution of low-dimensional data to identify outliers or leveraging deep neural networks to detect anomalies from high-dimensional inputs like video [ 1 ] . Approaches of the former category [ 2 , 3 , 4 , 5 , 6 , 7 ] perform well in unsupervised detection of deviations from proprioceptive measurements with sparse training data. For that, probabilistic distributions of expected measurements are learned from successful executions to detect deviations such as anomalous process forces with different thresholding methods. Deep learning-based detectors [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 ] utilize a supervised of self-supervised setting, typically requiring large training datasets of both positive and negative task examples, which makes them more effective at recognizing failures in environmental interactions, such as spilling liquid during pouring. Since the different detection methods vary in their training data requirements and computational complexity, it is beneficial to train individual detectors on distinct data subsets. However, to robustly detect anomalies in robotic manipulation tasks, a combined identification of deviations in motion, force, and interaction dynamics between the robot and its environment is required. Therefore, a comprehensive anomaly detection system must integrate multiple specialized subdetectors into a unified framework. We propose a novel multimodal anomaly detection strategy for robust task monitoring, integrating multiple detection approaches within a mixture-of-experts framework. Our method combines two complementary detectors: one analyzing the robot’s low-level dynamics and the other visually monitoring the scene to detect deviations from the expected task execution. A visual-language model (VLM) [ 8 ] detects violations of action preconditions and effects resulting from interaction with the environment, while a Gaussian-mixture regression (GMR)-based detector [ 2 ] continuously tracks deviations in interaction forces and robot poses. Considering the prediction confidence of each approach, their outputs are fused to produce a joint result. This late fusion approach allows the framework to dynamically select the most suitable detector in a given situation. Figure 1 highlights scenarios where either VLM-based or GMR-based anomaly detection is most effective, emphasizing the versatility and advantages of a fused approach. The contributions of our work are as follows: 1. a visual-language anomaly detection approach based on [ 8 ] which uses task progress to ground the expected action state and obtain a prediction confidence score; 2. a probabilistic anomaly detection method based on [ 2 ] using local modality-specific anomaly thresholds and quantification of prediction confidence; 3. a mixture-of-experts fusion strategy for merging probabilistic and deep-learning-based anomaly detection approaches. II Related works Previous works in robotics attempt to solve the anomaly detection problem by using either probabilistic modeling or deep learning. II-A Probabilistic anomaly detection Probabilistic approaches in [ 2 , 4 , 3 ] model successful task executions with Gaussian Mixture Models (GMM). By regressing the model on time, these methods compute expected sensor measurements and dynamically adjust anomaly detection sensitivity through probabilistic modeling. Chernova et al. [ 18 ] employ a GMM to encode a simple policy using basic symbolic actions and utilize the observation likelihood of unseen states for outlier detection. An offline anomaly detection method is presented in [ 5 ] , where Gaussian Processes Regression is used to predict force profiles and epistemic uncertainty during insertion tasks to compute an anomaly score at every time step. A run is unsuccessful if the average anomaly score over all time steps exceeds a predefined threshold. Probabilistic detection of anomalies can also be achieved with a Hidden Markov Model, either through predefined anomaly thresholds combined with a sliding time window approach [ 7 ] , or by estimating probabilistic thresholds based on execution progress [ 6 ] . These methods treat anomaly detection as statistical outlier detection of low-dimensional features derived from multimodal sensor data. However, unlike our VLM-based detector, they do not incorporate visual observations of the scene to identify anomalous effects of the robot’s actions on the environment. In [ 19 ] , the action consistency of a diffusion policy is evaluated across consecutive timesteps using a statistical distance function. An anomaly is detected when the cumulative sum of these distances exceeds a threshold learned from successful executions. This detector is combined with an LVLM to assess task progress through chain-of-thought reasoning and video question-answering, determining whether the robot is still progressing toward the task goal. However, unlike our approach, this method cannot detect deviations in contact force and exhibits significantly longer response times for the LVLM-based detector. II-B Deep-Learning anomaly detection Probabilistic approaches perform well with low-dimensional data, such as end-effector poses or force-torque measurements, but struggle with high-dimensional data like images or audio [ 1 ] . Deep-learning methods address this by learning feature representations while detecting anomalies [ 9 , 10 , 11 , 12 ] . Broadly, deep-learning anomaly detection techniques fall into four categories: reconstruction-based approaches, one-class neural networks, success detectors, and variants of visual- and large language models. Reconstruction-based methods treat anomaly detection as a self-supervised task, training generative models such as autoencoders [ 20 ] , variational autoencoders [ 9 ] , or generative adversarial networks [ 21 ] to encode and decode observations. Since they are trained only on “normal” data, they struggle to reconstruct anomalies, which results in

Read the complete text