Achieving Interaction Fluidity in a Wizard-of-Oz Robotic System: A Prototype for Fluid Error-Correction DOI: XXXXXXX.XXXXXXX Conference: Workshop on Errors, Mistakes, and Failures in Humans and Robots at 2026 ACM/IEEE International Conference on Human-Robot Interaction; March 16–19, 2026; Edinburgh, Scotland, UK ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing HCI theory, concepts and models CCS: Computing methodologies Robotic planning CCS: Computer systems organization External interfaces for robotics

Carlos Baptista De Lima, Julian Hough, Frank Förster, Patrick Holthaus, Yongjun Zheng · 2026

Read Achieving Interaction Fluidity in a Wizard-of-Oz Robotic System: A Prototype for Fluid Error-Correction DOI: XXXXXXX.XXXXXXX Conference: Workshop on Errors, Mistakes, and Failures in Humans and Robots at 2026 ACM/IEEE International Conference on Human-Robot Interaction; March 16–19, 2026; Edinburgh, Scotland, UK ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing HCI theory, concepts and models CCS: Computing methodologies Robotic planning CCS: Computer systems organization External interfaces for robotics by Carlos Baptista De Lima, Julian Hough, Frank Förster, Patrick Holthaus, Yongjun Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19374.txt.

Introduction

Baptista De Lima and colleagues set out rules for smooth interaction in a Wizard-of-Oz robot setup, where a hidden operator steers the robot while data is gathered. Speech-based robot interaction remains hard, and current use feels labored and frustrating. Their key rules cover interruptibility and correction, pollability, latency checks, and time-accurate replay from logs. They close by presenting a virtual reality test world for mobile manipulators that meets those rules.

Catalog record

CREATORCarlos Baptista De Lima, Julian Hough, Frank Förster, Patrick Holthaus, Yongjun Zheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23619
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19374

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Achieving Interaction Fluidity in a Wizard-of-Oz Robotic System: A Prototype for Fluid Error-Correction DOI: XXXXXXX.XXXXXXX Conference: Workshop on Errors, Mistakes, and Failures in Humans and Robots at 2026 ACM/IEEE International Conference on Human-Robot Interaction; March 16–19, 2026; Edinburgh, Scotland, UK ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Human-centered computing HCI theory, concepts and models CCS: Computing methodologies Robotic planning CCS: Computer systems organization External interfaces for robotics Carlos Baptista De Lima, Julian Hough, Frank Förster, Patrick Holthaus, Yongjun Zheng Abstract . Achieving truly fluid interaction with robots with speech interfaces remains a hard problem, and the experience of current Human-Robot Interaction (HRI) remains laboured and frustrating. Some of the barriers to fluid interaction stem from a lack of a suitable development platform for HRI for improving interaction, even in robotic Wizard-of-Oz (WoZ) modes of operation used for data collection and prototyping. Based on previous systems, we propose the properties of interruptibility and correction (IaC), pollability, latency measurement and optimisation and time-accurate reproducibility of actions from logging data as key criteria for a fluid WoZ system to support fluid error correction. We finish by presenting a Virtual Reality (VR) HRI simulation environment for mobile manipulators which meets these criteria. 1. Introduction The challenge of achieving fluid interaction remains prevalent within the field of human-robot interaction (HRI). Despite advances in robot vision, motion, manipulation and automatic speech recognition, state-of-the-art HRI is frequently slow, laboured, and fragile. Improving the feeling of fluidity is a difficult task to achieve, with a variety of challenges from the robot control algorithms to the physical constraints of the robots ( 10 ; 9 ; 1 ) . While fluidity has not been formally defined, it is felt by humans ( 10 ) . We define fluidity and fluid interaction as the ability to seamlessly transition in turn-taking to allow appropriate overlap of turns between agents, including multimodally, and to allow action using prediction. For example, in human-human pick-and-place situations, an instruction like “put the remote on the table" with a repair like “no, the left-hand table" typically has no delay in reacting to the initial instruction, and adaptation to the correction is instant. Fluid interaction for robots consists of more seamless, human-like transitions from processing speech to taking physical actions with no delays, permitting appropriate overlap between the two, and the ability to repair actions in real time as humans do ( 5 ) . While there has been some work on incremental processing for increasing the fluidity of HRI ( 10 ; 1 ) , to allow for the development of control algorithms by data-driven means, Wizard-of-Oz (WoZ) interactions, where an experimental confederate ‘Wizard’ tele-operates a robot out of view of the user, are required. However, it is not clear to what extent current WoZ systems are capable of achieving and recording data for developing fluid HRI, nor which properties are required to achieve this. After more precisely motivating our work and setting the contribution in context, we identify some core criteria of Wizard-of-Oz systems needed for interactive fluidity, briefly review existing WoZ-enabled robots and associated HRI experiments, specifically focusing on how they do or do not meet these criteria. We then present a prototype system in a VR environment which meets them. 2. Motivation & Related Work Table 1. An overview of existing HRI WoZ systems and their properties necessary for fluidity. Keys for the different interaction types: C = Controller, M = Mouse, K = Keyboard, P = Pen & Paper, H = Haptic/Touch, G = Gesture, T = Tablet Wizarding, S = Speech X = evidenced, ? = Mentioned category but not clearly discussed, - = Not Mentioned System Robot name Robot type Real or Virtual User interaction Wizard interaction Wizard interface Interruptibility and correction Pollability Reproducibility Latency ( 2 ) Pepper Social Robot VR H M+K Dashboard screen GUI - - - 45ms ( 8 ) UR10 Manipulator Arm RR P K Command-line interface X X - - ( 14 ) Pepper Social Robot RR ? M+K Dashboard screen GUI - - X ( 16 ) CityBot Mobile Car VR M+K+C G+S Dashboard screen GUI - - X X ( 15 ) Citybot Mobile Car RR S M Dashboard screen GUI ? - - - ( 19 ) Pepper Social Robot RR S + G M Dashboard screen GUI + Live Camera Feed - - - - ( 3 ) xArm 7 Manipulator Robot RR - C Dashboard screen GUI X - - X ( 24 ) Luxai QT robot Social robot RR P T Dashboard screen GUI - - - - ( 23 ) Luxai QT robot Social robot RR P T Dashboard screen GUI - - - - ( 7 ) Tiago++ Mobile Manipulator RR S M+K Dashboard screen GUI + Live Camera Feed - - - - ( 22 ) Multiple Social Robots RR - - - - - - - ( 20 ) ResolutionBot Social Robot RR H+S C+M+K Dashboard screen GUI + Live Camera Feed - - - ? ( 17 ) UR10 Manipulator Arm RR S+T M+K - - - - - ( 21 ) Nao Robot Social Robot RR S ? - - - - - Riek ( 13 ) provides a systematic review of 54 WoZ experiments published in the primary HRI publication venues from 2001–2011. They observed that research mainly used WoZ systems for simulating the natural language processing and non-verbal action functionality of systems (72.2% and 48.1% of papers, respectively). They found that Wizards also performed a significant amount of navigation and mobility tasks (29.6% of papers) while manipulation-based tasks were rarely employed (3.9% of papers). The author hypothesises this is largely due to the difficulty of fully automating speech-based systems as well as maintaining effective tracking. Two themes that occur throughout the reviewed papers are the concern of accurately representing the natural delay that robots have within the Wizarded systems, and accounting for errors from the Wizard, intentional or otherwise. Both themes are important when considering fluidity and the potential for repairing ongoing actions. One shortcoming of most current WoZ systems is that they tend to employ custom Graphical User Interfaces (GUIs) fit only for their specific problem with a finite set of potential actions ( 12 ) . The GUIs themselves can often lead to creating additional latency between a user’s command and the execution of the action, as found by Runzheimer et al ( 16 ) . 3. Background To understand the state of the literature on fluidity in Wizarded HRI systems based on the challenges discussed above, we conducted a systematic review. Our research question of the literature is this: To what extent is fluid interaction considered in the design of existing Wizard-of-Oz systems for robots? 3.1. Methodology The reason we exclude virtual agents in general is because of our domain of interest in robotics, however we still include virtual simulations of robots. The latter exclusion criterion mainly excluded non-peer-reviewed work on personal archive sites. After these selection criteria were applied, we arrived at 14 papers where our focus was on the properties of the WoZ system they describe. 3.1.1. Critical Appraisal The focus of the critical appraisal was on relevance (papers that reported an experiment, with a focus on using a

Read the complete text