MIHRaGe: A Mixed-Reality Interface for Human-Robot Interaction via Gaze-Oriented Control
Rafael R. Baptista, Nina R. Gerszberg, Ricardo V. Godoy, Gustavo J. G. Lahr · 2025
Read MIHRaGe: A Mixed-Reality Interface for Human-Robot Interaction via Gaze-Oriented Control by Rafael R. Baptista, Nina R. Gerszberg, Ricardo V. Godoy, Gustavo J. G. Lahr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03929.txt.
Introduction
Rafael Baptista and co-authors present MIHRaGe, a mixed-reality system that lets people steer a robot arm with their eyes. It targets users whose arms cannot move well. Gaze control exists, but users often cannot tell what the system thinks they picked. This design closes that loop with visual feedback inside a mixed-reality view.
Four participants tested the setup. Their gaze landed within about 1.46 centimeters of target on average. The arm itself placed objects with about 1.53 centimeters of error. In a pick-and-place task the system succeeded 80 percent of the time. The authors see real promise for daily assistance, with calibration and screen resolution as the next fixes.
Catalog record
| CREATOR | Rafael R. Baptista, Nina R. Gerszberg, Ricardo V. Godoy, Gustavo J. G. Lahr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03929
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MIHRaGe: A Mixed-Reality Interface for Human-Robot Interaction via Gaze-Oriented Control Rafael R. Baptista, Nina R. Gerszberg, Ricardo V. Godoy, Gustavo J. G. Lahr Abstract Individuals with upper limb mobility impairments often require assistive technologies to perform activities of daily living. While gaze-tracking has emerged as a promising method for robotic assistance, existing solutions lack sufficient feedback mechanisms, leading to uncertainty in user intent recognition and reduced adaptability. This paper presents the MIHRAGe interface, an integrated system that combines gaze-tracking, robotic assistance, and a mixed-reality to create an immersive environment for controlling the robot using only eye movements. The system was evaluated through an experimental protocol involving four participants, assessing gaze accuracy, robotic positioning precision, and the overall success of a pick and place task. Results showed an average gaze fixation error of 1.46 cm, with individual variations ranging from 1.28 cm to 2.14 cm. The robotic arm demonstrated an average positioning error of ≤ \leq 1.53 cm, with discrepancies attributed to interface resolution and calibration constraints. In a pick and place task, the system achieved a success rate of 80%, highlighting its potential for improving accessibility in human-robot interaction with visual feedback to the user. 1 Introduction Subjects that suffer from conditions that affect upper limb mobility, such as spinal cord injuries, neurodegenerative diseases, and strokes, often rely on external assistance due to the significant level of impairment to perform activities of daily living (ADLs) [ 1 , 2 ] . Assistive robotic systems have been developed to address this challenge, including exoskeletons, prosthetics, orthotics, and supernumerary robotic limbs, which can replace or augment lost motor functions [ 3 , 4 ] . These approaches leverage biosignals, including voice, head, or mouth movements, but most assistive robotic methods have inherent limitations, such as limb fatigue [ 5 ] and high susceptibility to background noise [ 6 , 7 ] . Eye tracking technologies have emerged as a promising solution, enabling real-time intention decoding and robot-actuated assistance through gaze-based control without dealing with limb fatigue and less background noise sensitivity [ 8 , 9 , 10 ] . Figure 1: Overview of the MIHRAGe interface: a user equipped with an eye tracker controls a robotic arm through a mixed-reality interface, which shows user control and robot intent. Several studies have integrated human-robot interaction with eye-tracking technologies to infer user intent and facilitate hands-free object manipulation. These systems leverage gaze-based selection mechanisms to generate movement commands, allowing users to interact with robotic arms and other assistive devices solely through eye movements [ 11 ] . Research has demonstrated the effectiveness of gaze-driven control across a wide range of applications, including assistive pick-and-place tasks, autonomous object retrieval, and remote robotic teleoperation, providing an alternative to conventional input methods that rely on residual motor function [ 12 , 13 , 14 ] . Despite these advancements, a significant gap remains in user experience limitations, particularly regarding the clarity of feedback in gaze-controlled robotic systems. One of the primary challenges is the lack of explicit feedback from the robot regarding its intended actions, which can lead to uncertainty in human-robot interaction [ 15 , 16 ] . Users often struggle to determine whether their gaze input has been correctly registered and interpreted by the system, which can reduce confidence and trust in its functionality [ 17 , 18 ] . Studies have shown that ambiguous system responses increase cognitive load, forcing users to repeat trial-and-error interactions to confirm the robot’s comprehension of their intent [ 19 ] . Recent research suggests that multimodal feedback mechanisms, such as augmented reality overlays or vibrotactile feedback, could enhance user awareness of robot state and intention, ultimately improving interaction efficiency and reducing user frustration [ 20 ] . However, the implementation of such features remains underexplored in gaze-based robotic control systems, highlighting the need for further research into optimizing feedback strategies to enhance user experience and system reliability. This paper presents the MIRAHGe interface, a novel approach that integrates gaze-tracking with a robotic arm and a mixed-reality interface. Unlike existing solutions, MIHRAGe provides real-time visual feedback on the gaze selection process and the robot’s intended actions, addressing the critical gap in user feedback mechanisms. By leveraging gaze-based control within a projected interface, the system allows users to control the robot’s position and execute pick and place with objects without requiring physical interaction. By dynamically overlaying relevant information onto the interface, such as gaze fixation points and robot movement intent, mixed reality helps users better anticipate and visualize system response. 1.1 Experimental setup The MIHRAGe interface uses a Tobii Pro Glasses 2 as the eye tracking device, which provides gaze coordinates in pixels at its frame, and also has a front-facing camera that was utilized to recognize the visual interface’s markers (AprilTags), assuming the user is always looking at the interface. The robotic arm integrated into the system was a Universal Robots UR20, a six-degree-of-freedom robot equipped with a Robotiq parallel gripper model 2F-140 for object manipulation. The interface, programmed using OpenCV-Python, is shown to the user using an Epson projector Powerlite E20. An overview of the experimental setup is shown in Fig. 1 . All the equipment is connected to a ROS2 network. The distance between the center of the markers is 70 cm \times 38.5 cm, and the area for interaction (i.e., the workspace) has 57 cm \times 25.6 cm. The interface was positioned 60 cm from the user to maintain comfortable visibility and effective gaze tracking. The system was powered by an Avell A52 laptop with processor i7-12650H, 32 GB of RAM, GPU GeForce RTX 3050, and Ubuntu 22.04. 1.2 Mixed Reality Interface The visual guide interface was developed to enable the user to control the movement of the robotic arm solely through gaze. This interface, shown in Fig. 2 , facilitates the selection of desired points within a predefined working area. It plays a crucial role by providing three main functionalities: user feedback, robot intent visualization, and environment interaction. Figure 2: An overview of the interface used to create the mixed-reality environment and the main frames. The interface frame is represented by \Sigma^{\prime} , the robot frame by \Sigma_{r} , and the user frame/eye tracker’s frame by \Sigma . The workspace for interaction is placed between the four markers (numbered from 0 to 3), along with the menu displayed with the option to choose PICK , MOVE , and CANCEL intention. The dashed lines, the frame orientation axis, and the marker numbering were included for illustrative purposes only. The user gazes at the interface while wearing the eye tracker. After two seconds of staring at a point, a cursor appears as a gray circle and remains static, marking the selected point. A menu appears around the circle with three options: MOVE , which commands the robotic arm to move above the selected point and remain stationary; PICK , positioning the robot above the desired point and proceeding to grasp the object using its parallel gripper; and CANCEL , used to cancel if the selected point is not of interest. If the PICK action is chosen, the next interaction cycle will replace this option with PLACE , allowing the user to place the object at another designated