REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery

1 st Giuliano Pioldi, 2 nd Yashika Batra, 3 rd Arman Ibrayeva, 4 th Yuanchen Bai, 5 th Purnjay Maruur, 6 th Promise Ekpo, 7 th Angelique Taylor · 2026

Read REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery by 1 st Giuliano Pioldi, 2 nd Yashika Batra, 3 rd Arman Ibrayeva, 4 th Yuanchen Bai, 5 th Purnjay Maruur, 6 th Promise Ekpo, 7 th Angelique Taylor (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29937.txt.

Introduction

Pioldi and six colleagues present REPAIR-Bench, a benchmark for how people perceive and recover from robot failures during interaction. Earlier work often treated failures as one-off events, scored them as simple success or failure, and modelled recovery with fixed rules.

The benchmark is built from 214 interaction trials with 41 participants, four induced failure types, and synchronised records of facial movements, head pose, speech, and after-session reports. Its three tasks cover detecting failures across linked sessions, classifying the failure type from vision, and predicting which recovery strategy the user prefers. A hierarchical recurrent model lifts failure detection strict F1 from 0.68 to 0.80 over a single-session model, and a tuned Mistral-7B reaches 0.76 on recovery prediction.

Catalog record

CREATOR1 st Giuliano Pioldi, 2 nd Yashika Batra, 3 rd Arman Ibrayeva, 4 th Yuanchen Bai, 5 th Purnjay Maruur, 6 th Promise Ekpo, 7 th Angelique Taylor
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery 1 st Giuliano Pioldi, 2 nd Yashika Batra, 3 rd Arman Ibrayeva, 4 th Yuanchen Bai, 5 th Purnjay Maruur, 6 th Promise Ekpo, 7 th Angelique Taylor Abstract Understanding how users perceive and respond to robot failures is essential for building robust and trustworthy robot systems. Prior work, however, (i) often treats failures as independent events, (ii) emphasizes binary failure detection, (iii) with rule-based recovery modeling. We present REPAIR-Bench, built on 214 interaction trials from 41 participants, the benchmark spans four induced failure types and provides synchronized facial action units, head pose, speech transcripts, and post-interaction affect and recovery reports. The benchmark spans three novel evaluation tasks that jointly capture the lifecycle of failure in human-robot interaction (HRI): (i) failure detection over inter-dependent interaction sessions, modeling longitudinal user adaptation across repeated failures; (ii) visual failure-type classification beyond binary success/failure formulations; and (iii) user-centered recovery prediction, inferring users’ preferred recovery strategies from interaction context rather than relying on manually designed or rule-based strategies. In baseline experiments, hierarchical recurrent modeling improved failure detection over a single-session model (strict F1: 0.80 0.80 vs. 0.68 0.68 ), achieved a failure localization mean signed error of − 0.51 -0.51 s, median absolute error of 2.97 s and, for recovery prediction, a QLoRA-tuned Mistral-7B reached Hit@ 5 = 0.76 5=0.76 and F1@ 5 = 0.32 5=0.32 . REPAIR-Bench provides both the HRI and Medical HRI communities with a standardized framework for (1) evaluating robot failures and (2) building transparent, adaptive, and trustworthy recovery systems. I Introduction Robots are increasingly developed for high-stakes environments, such as emergency departments [ 15 ] . However, even well-engineered systems are prone to failures in real-world settings, due to perception errors, time pressure, misinterpretation of user intent, or environmental uncertainty [ 6 ] . Characterizing failures and user responses is critical, as effective handling influences task completion, trust, and long-term acceptance of robots in the workplace. [ 6 ] . Fig. 1 : RFM-HRI data collection environment at an academic laboratory setting (left) and at the hospital (right) [ 13 ] . HRI research increasingly focuses on recognizing task failures through user behavior. System breakdowns disrupt interaction in measurable ways: the ERR@HRI dataset [ 4 ] highlights delays and robot-initiated interruptions as primary causes of rupture, while Kontogiorgos et al. [ 10 ] identify multimodal markers like gaze shifts and acoustic changes. Nonverbal cues often precede explicit verbal complaints, serving as early failure indicators [ 3 ] . Structured multimodal analysis has been advanced by the Response-to-Errors dataset, which uses OpenFace-extracted action units (AUs) [ 14 ] , and ERR@HRI 2.0, which includes corrective intent in conversational settings. Recent work leverages representation learning [ 12 ] , time-series pipelines [ 17 ] , and alignment strategies [ 8 ] to detect errors from user reactions alone. Further extensions have moved beyond technical failures to detect social breakdowns and repair attempts [ 11 ] , establishing a strong foundation for conversational failure detection. Despite these advances, several gaps remain. First, most benchmarks treat failures as independent samples rather than modeling cross-trial history or accumulation effects [ 9 ] . Second, existing datasets primarily support binary detection rather than multi-class failure categorization. Third, while prior work has examined recovery strategies, these are typically researcher-defined rather than predicted from user behavior. Finally, there is a lack of integrated benchmark frameworks connecting detection, understanding, and recovery decision-making within a single evaluation setting. Fig. 2 : REPAIR-Bench is multimodal framework for temporal robot failure detection, failure classification, and recovery strategy prediction. We introduce the Benchmark for Robot Error Perception And Interaction Recovery (REPAIR-Bench), the first HRI benchmark to jointly address failure classification, localization, and recovery prediction (Figure 2 ). Using the RFM-HRI dataset (Figure 1 ), we evaluate a Hierarchical Recurrent Neural Network (HRNN) for temporal detection and fine-tuned Small Language Models (SLMs) for recovery. Our results show that hierarchical modeling of cross-session history improves performance and that QLoRA-based instruction tuning is essential for structured output tasks. Contributions. REPAIR-Bench captures failures that occur while users interact with a physical robot to complete a physical retrieval task using a novel robot platform. We introduced three novel tasks, each designed to address key gaps mentioned above. The first task focuses on failure time detection and captures the accumulation of failure signals across multiple sessions. Importantly, it relies exclusively on visual behavioral signals, since they often provide earlier indicators of failure than verbal responses of users. In addition, audio inputs that include the robot’s speech would allow models to rely on the moment the robot begins speaking to infer when the failure occurred, while our focus is on capturing when and how external behavioral cues signal the emergence of a failure. The second task focuses on distinguishing between failure categories using solely visual cues of a user’s reaction, and provides critical contextual information about why the interaction broke down, which is essential for designing appropriate system responses. The third task predicts user-preferred recovery strategies using interaction transcripts and the failure context. The benchmark enables research on adaptive recovery behaviors that align with human expectations rather than designer-defined rules. Finally, REPAIR-Bench enables systematic investigation of how robots can detect failures, understand their underlying context, and select recovery strategies that help restore user confidence and trust after interaction breakdowns. In Lab Data Collection In Field Data Collection Fig. 3 : The RFM-HRI dataset [ 13 ] captures people performing object retrieval tasks with a medical robotic crash cart. CrashCart is a hospital-deployed mobile manipulator designed to support rapid crash-cart retrieval and handoff. It provides multimodal guidance via both visual and audio cues (e.g., light-emitting diode (LED) plus spoken instructions) to help users quickly understand the robot’s state and next actions during time-critical workflows. II Dataset and Preprocessing REPAIR-Bench builds on RFM-HRI, a multimodal dataset of HRIs during crash-cart item retrieval [ 13 ] , one of the only publicly available datasets that captures both robot failures and preferred recovery behaviors within a medical context. The dataset records interactions with a medical crash cart, a mobile cart used in hospital emergency rooms to store emergency medications and life-saving equipment for rapid access during critical events. The RFM-HRI dataset introduces a semi-autonomous crash-cart robot that guides users through retrieving specific items. The robot is a retrofitted medical crash cart equipped with LED indicators that highlight drawer locations, a text-to-speech interface for verbal guidance, and an egocentric camera that captures the interaction (Figure 3 ). The dataset contains 214 interaction samples from 41 participants collected in both laboratory and hospital settings. The original study included recordings from 41 participants, each completing five trials comprising four experimentally induced robot failure conditions in random order (i.e., speech failure, timing failure, comprehension failure, and search failure) and one error-free success condition (Figure 1 ). For each trial, the dataset includes synchronized video, audio, and task transcripts. Following each interaction, participants completed a post-interaction survey in which they reported (1) their emotional reactions to the trial and (2) their preferred recovery strategy (e.g., apology and clarification, step-by-step guidance, escalation to a human teammate). Each sample in the processed dataset includes: (i) facial Action Units (AUs) (18 intensity

Read the complete text