GEAR: Gaze-Enabled Human-Robot Collaborative Assembly
Asad Ali Shahid, Angelo Moroncelli, Drazen Brscic, Takayuki Kanda, Loris Roveda · 2025
Read GEAR: Gaze-Enabled Human-Robot Collaborative Assembly by Asad Ali Shahid, Angelo Moroncelli, Drazen Brscic, Takayuki Kanda, Loris Roveda (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-18947.txt.
Introduction
Asad Shahid and colleagues introduced GEAR in 2025, a gaze-enabled system that lets a robot help a person with assembly work. In their setup, the skilled worker does the fitting while the robot fetches parts. Voice or touch commands can slow that flow, so the robot instead watches where the user looks.
Thirty participants built assemblies of two complexity levels, using either gaze control or a touchscreen. With gaze, people finished with less physical demand and less effort, above all on the harder task. Performance stayed strong and parts arrived reliably. Participants also rated the gaze experience higher. The work points to eyes as a natural channel for guiding helper robots.
Catalog record
| CREATOR | Asad Ali Shahid, Angelo Moroncelli, Drazen Brscic, Takayuki Kanda, Loris Roveda |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-18947
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GEAR: Gaze-Enabled Human-Robot Collaborative Assembly Asad Ali Shahid, Angelo Moroncelli, Drazen Brscic, Takayuki Kanda, Loris Roveda Abstract Recent progress in robot autonomy and safety has significantly improved human-robot interactions, enabling robots to work alongside humans on various tasks. However, complex assembly tasks still present significant challenges due to inherent task variability and the need for precise operations. This work explores deploying robots in an assistive role for such tasks, where the robot assists by fetching parts while the skilled worker provides high-level guidance and performs the assembly. We introduce GEAR, a gaze-enabled system designed to enhance human-robot collaboration by allowing robots to respond to the user’s gaze. We evaluate GEAR against a touch-based interface where users interact with the robot through a touchscreen. The experimental study involved 30 participants working on two distinct assembly scenarios of varying complexity. Results demonstrated that GEAR enabled participants to accomplish the assembly with reduced physical demand and effort compared to the touchscreen interface, especially for complex tasks, maintaining great performance, and receiving objects effectively. Participants also reported enhanced user experience while performing assembly tasks. Project page: sites.google.com/view/gear-hri I Introduction In modern manufacturing, the assembly tasks involve numerous parts requiring precise operations across a wide range of product variants. Neither humans nor robots can efficiently complete the entire assembly process on their own. A collaborative approach leverages the complementary strengths of each partner: human dexterity and intelligence combined with robot precision. Such human-robot collaboration can enhance productivity, improve task quality, and reduce ergonomic strain on workers. For robots to effectively assist in assembly tasks, they need to accurately infer the user’s intentions. For example, if a robot anticipates that the user will assemble a specific part in the next step, it can proactively retrieve and deliver the part, minimizing the user’s idle time. In human communication, social cues, such as eye gaze, are fundamental for signaling engagement, interest, and attention. During manipulation tasks, individuals naturally focus their gaze on the object of interest before making a corresponding movement, typically shifting their gaze to the target object approximately 600 milliseconds before grasping it [ 1 ] . Similarly, in collaborative settings, such as object assembly, team members use gaze to indicate which tool or component they need from their teammates [ 2 ] . Eye gaze can therefore serve as a natural and intuitive signal for conveying a user’s intent to robots, potentially enabling more seamless collaboration. Fig. 1: GEAR enables users to request desired parts from the robot simply by looking at them while working on a primary task. The user’s intention is inferred by integrating their eye gaze with object detection on both the robot’s side (Fig. b) and the user’s side (Fig. c). The system is evaluated in two distinct collaborative assembly scenarios: assembling two sets of gears (Fig. a), and fastening screws while assembling a gear set (Fig. d). While gaze is promising for human-robot interaction (HRI), the investigation of gaze-based systems in collaborative assembly tasks is limited. This is partly due to the current limitations of gaze tracking technology and the challenges in effectively translating gaze information into robot actions. In particular, the 3D gaze estimation accuracy of most eye trackers is too low to achieve reliable robot control. For instance, if the robot is tasked with delivering parts to a user during an assembly based on their intention, it must be able to accurately determine the position of the parts. Although converting 3D gaze coordinates into the robot’s reference frame could theoretically indicate object locations, most eye trackers lack the precision for effective manipulation. To address these limitations, we introduce GEAR (see Fig. 1 ), a human-in-the-loop system that integrates gaze tracking and object detection to enhance human-robot collaboration in assembly tasks. The human provides a higher-level guidance by directing their gaze at a specific object, while the robot autonomously brings the object once it detects that the human is gazing at it. The system also incorporates auditory feedback to confirm the robot’s actions and guide users through the correct assembly sequence, ensuring smoother interaction. GEAR allows users to maintain focus on the primary task while intuitively requesting assistance from the robot. To evaluate the system’s efficacy, we conducted a user study with 30 participants, comparing the usability and performance of the gaze-based interface with that of a touch-screen-based interface. In summary, the main contributions of this work are as follows: • We develop a gaze-guided interaction system for human-robot collaborative assembly, combining gaze tracking with object detection alignment to enable intuitive, real-time collaboration without explicit commands. • We conduct a comprehensive user study comparing the usability and performance of our gaze-based system with a touch-based system, where users request objects via a touchscreen interface, evaluating both systems in terms of user workload, user satisfaction, and task efficiency in a collaborative assembly setting. II Related Works II-A Collaborative Assembly Systems Recent advances in human-robot collaboration have shifted from robots executing isolated sub-tasks [ 3 , 4 ] to intelligent partners that leverage perception and learning algorithms [ 5 , 6 , 7 ] . Several frameworks have been proposed for collaborative assembly—addressing task allocation [ 8 , 9 ] , scheduling [ 10 ] , and quality control [ 11 ] . In contrast to these approaches, our work specifically focuses on integrating gaze-based interactions into collaborative assembly systems. II-B Communication Interfaces Prior research in HRI has explored various communication modalities to convey human intentions to robots. Gesture-based interfaces, for instance, have shown significant potential, enhancing communication and improving task efficiency [ 12 , 13 ] . Gestures have been used to streamline tasks like vehicle door assembly [ 14 ] . On the other hand, auditory signals, such as speech-based communication [ 15 ] , can also be effective but often struggle in noisy environments. Eyes tend to move rapidly and serve both as a means of gathering and conveying information. This makes eye-tracking an effective way to convey the intentions to other agents [ 16 , 17 ] . Prior studies have shown that gaze cues can improve shared attention and fluency during handovers [ 18 ] , and provide contextual cues that enhance the robot’s understanding of human intentions [ 16 ] . Eye gaze tracking has also been used to enhance imitation learning in robotic manipulation by discarding task-irrelevant objects [ 19 ] and to assess the operator’s comfort level during a collaboration [ 20 ] . In the realm of assistive robotics, gaze has been used to improve robotic rehabilitation, reducing workload for the user [ 21 , 22 ] . The authors in [ 23 ] proposed a system that enhances user input via a joystick with autonomous functionalities of the robot, effectively inferring user intentions through eye gaze tracking. Despite advancements in gaze-based collaborative systems, leveraging eye gaze for collaborative assembly remains notably limited. While gaze has been used to infer user intention in prior research, there is a lack of studies utilizing gaze for real-time parts retrieval in human-robot assembly systems. This study addresses this gap by developing and evaluating a gaze-guided interaction system that integrates eye gaze tracking with object detection on both the robot and the user side, enabling intuitive