ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Computer systems organization Robo

Shiye Cao, Maia Stiber, Amama Mahmood, Maria Teresa Parreira, Wendy Ju, Micol Spitale, Hatice Gunes, Chien-Ming Huang · 2025

Read ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Computer systems organization Robo by Shiye Cao, Maia Stiber, Amama Mahmood, Maria Teresa Parreira, Wendy Ju, Micol Spitale, Hatice Gunes, Chien-Ming Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-13468.txt.

Introduction

Shiye Cao and co-authors launch a challenge in 2025 for spotting robot errors in conversation. Language-model robots talk more freely than ever, yet they still mishear intent, interrupt, or go silent. Catching those failures early protects tasks and user trust.

The challenge ships sixteen hours of human-robot dialogue with face, speech, and head movement data. Each clip is labeled for robot errors and for moments where users meant to correct the machine. Teams are invited to build detectors from the multimodal signals. Scoring weighs detection accuracy against false alarm rates. The set is the second edition of a growing benchmark series.

Catalog record

CREATORShiye Cao, Maia Stiber, Amama Mahmood, Maria Teresa Parreira, Wendy Ju, Micol Spitale, Hatice Gunes, Chien-Ming Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-13468

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ERR@HRI 2.0 Challenge: Multimodal Detection of Errors and Failures in Human-Robot Conversations DOI: XXXXXXX.XXXXXXX Conference: Proceedings of the ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland ISBN: 978-1-4503-XXXX-X/2025/06 CCS: Computer systems organization Robotics CCS: Computing methodologies Artificial intelligence CCS: Human-centered computing Human computer interaction (HCI) Shiye Cao, Maia Stiber, Amama Mahmood, Maria Teresa Parreira, Wendy Ju, Micol Spitale, Hatice Gunes, Chien-Ming Huang Abstract . The integration of large language models (LLMs) into conversational robots has made human-robot conversations more dynamic. Yet, LLM-powered conversational robots remain prone to errors, e.g., misunderstanding user intent, prematurely interrupting users, or failing to respond altogether. Detecting and addressing these failures is critical for preventing conversational breakdowns, avoiding task disruptions, and sustaining user trust. To tackle this problem, the ERR@HRI 2.0 Challenge provides a multimodal dataset of LLM-powered conversational robot failures during human-robot conversations and encourages researchers to benchmark machine learning models designed to detect robot failures. The dataset includes 16 hours of dyadic human-robot interactions, incorporating facial, speech, and head movement features. Each interaction is annotated with the presence or absence of robot errors from the system perspective, and perceived user intention to correct for a mismatch between robot behavior and user expectation. Participants are invited to form teams and develop machine learning models that detect these failures using multimodal data. Submissions will be evaluated using various performance metrics, including detection accuracy and false positive rate. This challenge represents another key step toward improving failure detection in human-robot interaction through social signal analysis. Figure 1. In this challenge, we invite participants to develop multimodal ML models designed to detect conversational robot failures using a dataset that includes facial, head pose, and speech features extracted from 16 hours of dyadic human-robot conversations multimedia. 1. Introduction In recent years, the integration of large language models (LLMs) into conversational robots—embodied media—has enabled more natural human-robot conversations. However, while LLMs can mitigate certain speech recognition errors that often cause traditional conversational robots to fail, these advanced systems remain vulnerable to issues such as interrupting users and misinterpreting user intent ( Mahmood et al. 2025b ; Mahmood et al. 2024 ; Cao et al. 2025 ) . Such failures can lead to conversational breakdowns, disrupt the interaction flow, and negatively affect the user’s perception of the robot. As LLM-powered conversational robots are being deployed in real-world scenarios, the need to understand and mitigate their failures is a pressing issue. Previous works have discovered some strategies to help robots rebuild user trust and recover from errors ( Karli et al. 2023 ; Esterwood and Robert 2021 ) ; however, the application of these strategies hinges on the robot’s ability to detect its own errors effectively ( Stiber et al. 2023 ) . Status quo robot error detection methods often either depend on task-specific or domain-specific information—which can be hard to generalize across people, tasks, and errors—or rely on users to detect and report errors—which can be delayed. Recent works showed the possibility of detecting robot error through leveraging social signals exhibited by user reactions and responses to robot errors in multimedia streams ( Stiber et al. 2022 ; Stiber et al. 2025 ; Bremers et al. 2023 ; Spitale et al. 2023 ) . To further explore this potential and benchmark multimodal robot failure detection in HRI, the ERR@HRI 2.0 challenge focuses on the multimodal detection of robot errors and failures in dyadic human-robot conversations . The ERR@HRI initiative serves as a platform for benchmarking not only HRI datasets but also machine learning models aimed at detecting robot errors and failures from multimedia. The 2024 ERR@HRI challenge, held as part of the 2024 ACM International Conference on Multimodal Interaction (ICMI’24), focused on modeling robot errors using a multimodal dataset collected in a real-world setting where a robotic coach delivered well-being coaching practices to each individual separately over four weeks ( Spitale et al. 2024 ) . Building on this, the ERR@HRI 2.0 challenge uses a new multimodal dataset (see Fig. 1 ) comprised of multimodal behavioral signals extracted from audio and video recordings of human conversations with two distinctive LLM-powered robots: a social robot with more anthropomorphic features and expressions, and a smart speaker. The user and the robot collaborated on five tasks during the conversations, ranging from medical self-diagnosis, trip planning, selecting items more beneficial during a desert survival simulation, to discussing whether the federal government should ban capital punishment and whether universities should have their own police force. Additionally, the ERR@HRI 2.0 challenge is also unique in that it contains two sub-challenges to tackle robot error detection from two perspectives. Sub-challenge 1 targets the detection robot errors and failures from the system perspective, while sub-challenge 2 targets the detection of robot errors and failures from the user perspective. Through this challenge, we hope to bring together researchers and practitioners from the multimedia community and the robotics and human-robot interaction community, while highlighting the underlying multimedia aspects of the problem. We invite teams to develop and benchmark multimodal machine learning algorithms to detect robot errors and failures in a human-robot conversation dataset, with multimodal features and ground truth labels gathered from approximately 16 hours of multimedia. The techniques developed through this challenge will contribute to advancing our understanding of how multimedia can enhance the analysis and interpretation of interactions between humans and autonomous robots. 2. Related Work 2.1. Robot Errors As robots become more capable, they are increasingly being integrated into daily life, assisting and collaborating with human users across a range of tasks and contexts ( Mahmood et al. 2025a ; Gerling et al. 2016 ; Portugal et al. 2019 ; Scassellati et al. 2018 ) . However, errors and failures remain common and often inevitable in real-world deployed robots ( Carlson and Murphy 2005 ; Honig and Oron-Gilad 2018 ; Mirnig et al. 2015 ) . For a robot to be truly effective in a human-robot team, it must not only complete its assigned tasks but also adapt to the user and align its behavior with the user’s expectations—thereby enhancing team performance and user satisfaction ( Tabrez et al. 2020 ) . This highlights a limitation in existing robot evaluation approaches that only assess the robot based on task completion. Instead, we need to consider how the task was performed and whether the robot’s behavior conformed to their human teammate’s expectations. Therefore, in this work, we adopt a broader definition of robot errors that assesses the robots behavior from two perspective: 1) robot errors from the system perspective —deviations in the robot behavior from its expected behavior by system design; 2) robot errors from the user perspective —deviations in the robot behavior from the user’s mental model of the expected robot behavior. 2.2. Robot Error Detection Using Social Signals Social signals are multimodal behavioral cues (both verbal and non-verbal) that can convey information about one’s emotions, intentions, attitudes, and social dynamics within interactions ( Vinciarelli et al. 2008 ; Cao and Huang 2022 ; Candon et al. 2023 ; Sauppé and Mutlu 2014 ; Stiber et al. 2024 ) . In human-robot interaction (HRI), users’ natural responses (social signals) during the interaction can impart information about their own internal state ( e.g., user uncertainty) and their

Read the complete text