RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation * These authors contributed equally to this work.

First A. Zitiantao Lin and Yongpeng Sang. Member, IEEE, Zitiantao Lin *, Yongpeng Sang *, Yang Ye · 2025

Read RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation * These authors contributed equally to this work. by First A. Zitiantao Lin and Yongpeng Sang. Member, IEEE, Zitiantao Lin *, Yongpeng Sang *, Yang Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22113.txt.

Introduction

Zitiantao Lin, Yongpeng Sang, and Yang Ye let a robot arm take orders from a glance. Robot helpers fetch objects for people with mobility impairments. Joysticks are the usual control, but they demand precise hand control. They also force users to think in the machine's frame, not their own.

Their system, RaycastGrasp, pairs a wearable mixed reality headset with the arm. The user simply looks at the wanted object. The headset reads the gaze fixation from a first-person view. Augmented cues confirm the intent. A pretrained vision model and the arm then carry out the grasp. Experiments showed clear gains over conventional control. The authors position eye guidance as a more accessible front end for assistive robots.

Catalog record

CREATORFirst A. Zitiantao Lin and Yongpeng Sang. Member, IEEE, Zitiantao Lin *, Yongpeng Sang *, Yang Ye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RaycastGrasp: Eye-Gaze Interaction with Wearable Devices for Robotic Manipulation * These authors contributed equally to this work. First A. Zitiantao Lin and Yongpeng Sang. Member, IEEE, Zitiantao Lin *, Yongpeng Sang *, Yang Ye Abstract Robotic manipulators are increasingly used to assist individuals with mobility impairments in object retrieval. However, the predominant joystick-based control interfaces can be challenging due to high precision requirements and unintuitive reference frames. Recent advances in human-robot interaction have explored alternative modalities, yet many solutions still rely on external screens or restrictive control schemes, limiting their intuitiveness and accessibility. To address these challenges, we present an egocentric, gaze-guided robotic manipulation interface that leverages a wearable Mixed Reality (MR) headset. Our system enables users to interact seamlessly with real-world objects using natural gaze fixation from a natural first-person perspective, while providing augmented visual cues to confirm intent and leveraging a pretrained vision model and robotic arm for intent recognition and object manipulation. Experimental results demonstrate that our approach significantly improves manipulation accuracy, reduces system latency, and achieves single‑pass intention and objection recognition accuracy > > 88% across multiple real-world scenarios. These results demonstrate the system’s effectiveness in enhancing intuitiveness and accessibility, underscoring its practical significance for assistive robotics applications. I Introduction Although joystick-based control remains the most common approach for wheelchair robots and industrial manipulators [ 1 ] [ 2 ] , this method has notable drawbacks, including high precision requirements, unintuitive reference frames, and substantial cognitive and physical demands on users. In practice, users must repeatedly adjust joystick direction and force, which can lead to fatigue and reduced operational efficiency—especially for individuals with limited hand dexterity or motor impairments. To address these challenges, recent years have seen the emergence of alternative control methods such as reference frame reallocation [ 3 ] , gesture control, and vision-based interfaces. However, these solutions often require extensive training, lack intuitive feedback, or suffer from reliability issues [ 1 ] . Gaze-based robotic interaction has thus emerged as a promising alternative, enabling intuitive, hands-free operation through natural gaze behavior. Currently, various eye-gaze-based control systems [ 4 ] have been developed within the Human-Robot Interaction (HRI) framework [ 5 ] . One of the earliest applications utilizing eye tracking for creative tasks was EagleEyes [ 6 ] , a control system enabling individuals with disabilities to draw on a screen. Another example is the GazeGrasp system [ 7 ] , which enables robotic grasping by interpreting users’ gaze on a computer screen. However, most of these systems rely on a third-party screen as the interaction medium, using the robot’s perspective as the primary viewpoint for manipulation. Alternatively, some emerging eye-gaze models employ head-mounted eye trackers, but these approaches often constrain the user’s head posture or rely on supplementary optical tracking devices, such as OptiTrack, to accurately capture head movements. To overcome these challenges, our approach emphasizes egocentric, attention-based interaction, enabling users to intuitively select and manipulate real-world objects through natural gaze fixation in a Mixed Reality (MR) environment. By leveraging gaze as an explicit indicator of user intent, the system supports hands-free, first-person HRI. To realize this, we design a system that seamlessly integrates a Franka Emika collaborative robotic arm, a wearable MR headset, and a self-developed eye-tracking algorithm. The user’s gaze, captured from their own perspective, is used to identify and select objects in the environment. Object recognition is performed via a pretrained YOLOv8 deep learning model, while the robotic arm executes the corresponding manipulation. This design bridges immersive, user-centered MR experiences with practical robotic assistance, showcasing the power of intuitive, gaze-driven control. Experimental results show that this approach significantly improves manipulation accuracy, reduces system latency, and enhances user satisfaction compared to traditional screen-based eye-tracking methods. The system achieves a single‑pass intention and objection recognition accuracy confidence of > 88% across multiple real-world scenarios, demonstrating its robustness and versatility. II Related Work II-A Gaze Fixation as a Cognitive and Social Signal Gaze fixation, defined as the sustained visual attention directed at a particular object or spatial location, serves as a crucial indicator of both cognitive processing and social intention [ref]. In instructional contexts, teacher gaze patterns have been shown to reflect specific pedagogical goals: fixating on students is often used to guide attention, on instructional content to emphasize key information, and on blank spaces to indicate cognitive effort such as planning or hesitation [ref]. These patterns suggest that both the duration and spatial target of a fixation can reveal an individual’s cognitive load and communicative intent [ 8 ] . In the domain of HRI, fixation duration plays a similarly critical role in how users interpret the robot’s intent. Short glances (less than 1 second) are typically perceived as unintentional or non-interactive, while prolonged fixations (exceeding 2 seconds) are more likely interpreted as signals of interest, inquiry, or social engagement [ 9 ] . These findings highlight the importance of modeling fixation duration when designing gaze-based interactive systems. Beyond temporal features, gaze fixation is also central to semantic inference and intention understanding. Neurophysiological evidence shows that observing another’s gaze toward an object activates brain regions akin to those triggered by observing goal-directed actions like reaching or grasping. This supports the view that gaze serves as a non-verbal proxy for intention communication [ 10 ] . Fixation has also been shown to strongly correlate with attention allocation and cognitive effort. For example, in natural tasks, individuals tend to fixate on objects before acting on them, and shift gaze only once the action is nearly complete [ 10 ] . Moreover, gaze aversion—momentarily looking away—is often employed to regulate cognitive processing or reduce interpersonal stress, depending on the duration and context of the fixation [ 10 ] . This behavior and neurological studies provide solid theoretical scaffolding for gaze-based intention prediction that can be applied in assistive robots. II-B Gaze-Based Robotic Interaction in MR and HRI Gaze behavior is a crucial interaction modality in both MR and HRI, as it reflects the user’s real-time focus of attention and underlying intention. In MR environments, Baptista et al. [ 11 ] proposed an interaction system called MIHRaGe, which enables users with limited limb mobility to control a robotic arm solely through eye movements. When a user fixates on a specific point on the interface, a menu with four options—MOVE, PICK, PLACE, and CANCEL—is triggered, allowing for contactless robotic control. The system achieves a task success rate of 80%, using a 2-second fixation as the selection trigger. However, the system requires that all target objects be pre-placed within a predefined workspace, which limits its adaptability to real-world environments. To address this limitation, Wang et al. [ 12 ] developed a gaze-driven robotic grasping system that combines near-eye tracking with GraspFormer, a transformer-based grasp candidate generator. By estimating 3D gaze vectors and aligning them with generated grasp candidates, the system achieves over 95% grasping accuracy with a response latency below 100 ms. However, as the interaction still occurs on a 2D image plane, it lacks true MR spatial anchoring, which may hinder depth perception and reduce interaction naturalness. Furthermore, Choi et al. [ 13 ] introduced an MR-based HRI system for infrastructure inspection that classifies gaze behavior into scanning, focusing, and inspecting categories. It employs a virtual

Read the complete text