Failure Identification in Imitation Learning via Statistical and Semantic Filtering

Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret · 2026

Read Failure Identification in Imitation Learning via Statistical and Semantic Filtering by Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13788.txt.

Introduction

Rolland, Mayran de Chamisso, and Mouret build FIDeL, a module that spots genuine failures in imitation-learned robot policies. Rare faults, defective parts, or unexpected human actions can break executions. Anomaly detectors often cannot tell failures from harmless deviations. FIDeL represents nominal demos compactly. It aligns new observations by optimal transport matching to score anomalies and map them. A vision-language model then filters benign anomalies by meaning. On the new BotFails dataset it gains over 5 percent in AUROC and over 17 percent in failure-detection accuracy.

Catalog record

CREATORQuentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34886
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13788

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Failure Identification in Imitation Learning via Statistical and Semantic Filtering Quentin Rolland, Fabrice Mayran de Chamisso, Jean-Baptiste Mouret Abstract Imitation learning (IL) policies in robotics deliver strong performance in controlled settings but remain brittle in real-world deployments: rare events such as hardware faults, defective parts, unexpected human actions, or any state that lies outside the training distribution can lead to failed executions. Vision-based Anomaly Detection (AD) methods emerged as an appropriate solution to detect these anomalous failure states but do not distinguish failures from benign deviations. We introduce FIDeL (Failure Identification in Demonstration Learning), a policy-independent failure detection module. Leveraging recent AD methods, FIDeL builds a compact representation of nominal demonstrations and aligns incoming observations via optimal transport matching to produce anomaly scores and heatmaps. Spatio-temporal thresholds are derived with an extension of conformal prediction, and a Vision–Language Model (VLM) performs semantic filtering to discriminate benign anomalies from genuine failures. We also introduce BotFails , a multimodal dataset of real-world tasks for failure detection in robotics. FIDeL consistently outperforms state-of-the-art baselines, yielding +5.30% AUROC in anomaly detection and +17.38% failure-detection accuracy on BotFails compared to existing methods. Videos of FIDeL can be found on our website : https://cea-list.github.io/FIDeL/ I INTRODUCTION Recent progresses in imitation learning (IL) [ 22 , 7 , 53 ] have the potential to significantly enhance the flexibility and adaptability of robotic systems. Unfortunately, current learned policies cannot be deployed in real world environments because their behavior is not defined when they are in a situation that is not part of the training set. Such events are unavoidable in factories or human-centered environments, arising from defective objects, operator mistakes, or environmental shifts, and they cannot be exhaustively anticipated in training data. A prevalent strategy is to assume that the training dataset is sufficiently diverse to encompass all possible scenarios. Implicitly, this amounts to presuming that no situation is off distribution. While this assumption simplifies the problem, it is unrealistic in practice, many rare events will never appear in training data. The second approach is to design models that can quantify the uncertainty of their decision, for instance with Bayesian models [ 14 , 23 ] . However, uncertainty quantification remains an open challenge in IL and RL [ 30 , 55 ] , and is not integrated into the highest-performing learning algorithms [ 22 , 7 , 53 ] . A more practical direction is to rely on an independent monitoring module that evaluates execution at runtime [ 35 , 13 , 18 , 2 , 49 ] . In this paper, we build on recent work [ 35 , 13 , 18 , 2 , 49 ] and propose an independent failure monitor, designed to detect situations in which a policy should be interrupted. We frame the problem in terms of Anomaly Detection (AD). Anomalies correspond to deviations from the expected distribution, which may—but do not always—indicate failure states. Our approach leverages the One-Class (OC) paradigm, a well-established method in computer vision [ 9 , 1 , 47 , 5 , 41 ] . In OC learning, a model is trained solely on data from a “normal” class, and any significant deviation from this baseline is flagged as anomalous. This formulation is particularly well-suited to IL, where the space of possible deviations is unbounded, yet expert demonstrations naturally provide a reliable definition of normality. Thus, we treat expert demonstrations as our baseline and regard any deviation as an anomaly—yielding an implicit and comprehensive definition of undesirable behavior. That said, the notion of anomaly encompasses a broader range of situations than failure. Not all anomalies correspond to failure-related events: some may be entirely benign—for instance, a fly landing on a table or an object slightly shifting in the background—and should not trigger unnecessary or costly interventions. This mismatch reveals a key limitation of existing AD methods: while they can reliably highlight deviations from nominal behavior, they lack the ability to discriminate between harmless anomalies and those that truly compromise task execution. In robotics, this distinction is essential. What is required is a mechanism that can semantically interpret anomalies and explicitly determine whether they represent benign variations or genuine failures. To bridge the gap between traditional Vision Anomaly Detection and the unique demands of robotic applications, we introduce FIDeL ( F ailure I dentification in De monstration L earning), a failure detection module designed to complement IL policies in robotics. FIDeL performs post-hoc anomaly detection by comparing new observations with stored representations of normal demonstrations using Optimal Transport (OT). To decide whether or not an AD score indicates an anomaly, we leverage a Conformal Prediction (CP)–based thresholding [ 25 , 12 ] mechanism, which accounts for both temporal variations in anomaly scores and spatial variations across visual patches. When the anomaly score exceeds the CP threshold, FIDeL triggers a semantic filter based on a vision-language model (VLM), which determines whether the detected anomaly is failure-related or benign. This three-stage process allows for efficient failure detection while reducing false positives, thus preserving productivity. Our contributions are summarized as follows: • Following the One-Class paradigm, we design a novel Vision Anomaly Detection, representation-based algorithm tailored for autonomous robotics. • We extend Conformal Prediction to handle both temporal and spatial variations , with memory-based temporal alignment that ensures robustness to variable execution speeds. • We propose a semantic filtering mechanism using a VLM and anomaly heatmaps, which discriminates between benign anomalies and task-critical failures. • We enhance interpretability through localized heatmaps and VLM-based explanations , providing human-readable justifications for failure detection. • We present BotFails a dedicated training and evaluation dataset collected on LeRobot [ 4 ] for benchmarking anomaly and failure detection in robotics, addressing the scarcity of failure-oriented datasets. • We demonstrate significant performance gains over baselines in both anomaly detection ( +5.30\% AUROC) and failure detection tasks ( +17,38\% accuracy). II RELATED WORK II-A Vision Anomaly Detection Vision AD is a well-established problem in computer vision, particularly within the unsupervised learning paradigm. Existing methods can be broadly categorized into four families [ 9 ] : (1) Augmentation-based methods add artificial anomalies to normal samples and train a classifier to recognize them [ 26 , 52 , 38 ] . However, we found no straightforward way to adapt these approaches to robotics, primarily due to the challenge of generating realistic anomalies in robot and object motions. (2) Reconstruction-based methods learn to reconstruct normal inputs, using the reconstruction error as an anomaly score. Common architectures include Autoencoders (AE), Variational Autoencoders (VAE) or Generative Adversarial Networks (GANs) [ 42 , 29 , 39 ] , and Student-Teacher frameworks [ 43 , 50 , 37 ] . However, such models often generalize well enough to reconstruct anomalous inputs, thus failing to highlight anomalous regions effectively [ 8 , 3 ] . (3) Normalizing Flows (NF) [ 34 ] based methods [ 16 , 36 , 51 ] learn an invertible transformation from the data distribution to a well-defined prior, typically a Gaussian. Anomalies are detected when the likelihood of a test sample under the learned distribution deviates significantly from the normal class. Despite their theoretical appeal, NF exhibit “likelihood paradoxes,” often assigning higher likelihood to anomalous images than normal ones [ 24

Read the complete text