Hybrid Attention Estimation Pipeline for Adaptive HRI Using an Expressive Robotic Head

Pablo Moraes 1, Mónica Rodríguez 1, Christopher Peters 2, Hiago Sodre 1, Tobias Doernbach 2, Bruna Guterres 1, and Ricardo Grando 1∗ · 2026

Read Hybrid Attention Estimation Pipeline for Adaptive HRI Using an Expressive Robotic Head by Pablo Moraes 1, Mónica Rodríguez 1, Christopher Peters 2, Hiago Sodre 1, Tobias Doernbach 2, Bruna Guterres 1, and Ricardo Grando 1∗ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2608-00284.txt.

Introduction

Moraes and co-authors present a hybrid attention pipeline for human-robot interaction. Their case study uses an expressive robotic head from the InMoov ecosystem. One layer is fast and geometric in form. It gives high-frequency face and head-pose data for timing control. A second layer is semantic, built on a vision-language model.

The semantic layer sees only raw camera frames from the robot. It labels attention toward the robot, phone use, or attention elsewhere. A finite state machine merges both signal streams. It governs activation, waiting, resumption, and return to rest. Ten participants ran 40 trials across baseline and adaptive conditions. Interaction starts were reliable in every trial. Pause behaviour in the distraction condition was consistent. The two layers carried non-redundant information, they report.

Catalog record

CREATORPablo Moraes 1, Mónica Rodríguez 1, Christopher Peters 2, Hiago Sodre 1, Tobias Doernbach 2, Bruna Guterres 1, and Ricardo Grando 1∗
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2608.00284

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hybrid Attention Estimation Pipeline for Adaptive HRI Using an Expressive Robotic Head Pablo Moraes 1, Mónica Rodríguez 1, Christopher Peters 2, Hiago Sodre 1, Tobias Doernbach 2, Bruna Guterres 1, and Ricardo Grando 1∗ Abstract This paper presents an applied case study on hybrid visual attention estimation for human–robot interaction using an expressive robotic head based on the InMoov ecosystem. The proposed pipeline combines a fast geometric perception layer with an independent semantic perception layer based on a vision–language model. The geometric layer provides high-frequency face and head-pose information for temporal regulation, while the semantic layer receives only raw egocentric camera frames and produces contextual attention labels related to attention toward the robot, phone use, or attention elsewhere. These signals are integrated through a finite state machine that regulates adaptive interaction behavior, including activation, waiting, interaction resumption, and return to rest. The system was evaluated with 10 participants across 40 trials covering baseline and adaptive interaction conditions. Results show reliable interaction start across all trials, consistent pause behavior in the adaptive distraction condition, and non-redundant semantic information between the geometric and semantic outputs. A demonstration video of the system is available at: https://youtube.com/shorts/jbKdYOb6ezs?si=JfecAN6WGFC9BrQH . \affils 1 Technological University of Uruguay, Uruguay 2 Ostfalia University of Applied Sciences, Germany ∗ Corresponding author: [email protected] keywords Human–robot interaction, visual attention estimation, expressive robotic head, vision–language model, social robotics, state machine 1 Introduction In social contexts, human–robot interaction (HRI) requires robots to generate verbal responses and to perceive non-verbal user cues, adapting their behavior in a visible and coherent manner. Visual attention is a key cue that enables the system to infer user availability, intention, and engagement. For robots with expressive heads, head orientation, eye contact, and facial expressions are particularly important channels for conveying the system’s internal state and willingness to interact [ 1 , 2 ] . It has been reported that mechanisms such as eye contact and joint attention play a structural role in social interaction, fostering coordination between human and robot and improving the fluency of the exchange [ 3 , 4 , 5 ] . Likewise, visual attention estimation has been addressed through cues such as face detection, head pose, body orientation, gaze direction, and contextual information, showing that attention should be understood as a dynamic and multimodal state rather than as an isolated binary variable [ 6 , 7 , 8 , 9 ] . However, much of the literature addresses attention estimation, expressive cue design, and physical platform implementation separately. Although accessible HRI platforms and low-cost expressive robotic heads have been proposed, a gap still remains between visual attention perception and its systematic translation into embodied expressive behaviors within a reproducible and physically accessible robotic head [ 10 , 11 , 12 ] . In this context, the InMoov ecosystem is particularly relevant as an open and reproducible basis for the development of humanoid and animatronic platforms oriented toward experimentation [ 13 , 14 ] . In addition, although vision–language models offer new possibilities for interpreting visual context in social robots, their integration into embodied real-time HRI systems remains challenging [ 15 ] . Figure 1 : Expressive robotic head based on the InMoov open-source platform. This work presents an applied case study on hybrid visual attention estimation for HRI using an expressive robotic head based on InMoov. The proposed pipeline combines a fast geometric perception layer for user presence and attention estimation with an independent semantic layer for contextual interpretation. Based on this information, the system generates discrete social states that regulate observable robot behaviors such as activation, waiting, resuming, and returning to rest. The main contribution of this work is a hybrid attention-estimation pipeline that connects visual attention estimation, contextual interpretation, and adaptive interaction behavior in a physical robotic platform. The proposed system improves the interaction pipeline by combining a high-frequency geometric attention layer for real-time state regulation with an independent VLM-based semantic observer, enabling attention-aware pause and resume behaviors during human–robot interaction. The pipeline is evaluated through objective metrics of interaction start reliability, protocol outcomes, response timing, and geometric–semantic divergence. To support reproducibility, the open-source implementation is freely available on https://github.com/ShiiSuii/ICCAS_STUDY . 2 Related Work Visual attention and social gaze have been widely studied as central components of human–robot interaction. Joint attention mechanisms allow humans and robots to coordinate their focus during interaction, improving task performance, social presence, and perceived robot competence [ 5 , 16 , 17 ] . More broadly, Admoni et al. [ 1 ] organize social gaze in HRI from human-centered, design-centered, and technology-centered perspectives, highlighting its relevance both as a perceptual cue and as an expressive behavior. Several works have addressed visual attention estimation from observable user cues such as face detection, head pose, gaze direction, body orientation, and facial expression. Veronese et al. [ 7 ] showed that head pose can be used as a practical signal for estimating attentional direction, while Li et al. [ 6 ] proposed a vision-based system that combines multiple cues for natural interaction in open environments. Attention has also been used as a control variable for robot behavior. Das et al. [ 8 , 18 ] proposed vision-based attention control mechanisms that regulate awareness signals, eye contact, and attentional redirection. Similarly, Oishi et al. [ 19 ] modeled robot behavior through a finite state machine adapted to the perceived state of the person. These works are closely related to the present study, since attention is not only estimated, but also used to regulate visible interaction states. Expressive robotic heads provide suitable physical platforms for studying this type of interaction, since they combine perception, head orientation, and facial expressiveness in a compact embodied system. Rojas-Quintero et al. [ 2 ] review sensor heads for humanoid robots and distinguish between expressive and non-expressive platforms. Other works have proposed accessible or open robotic heads for HRI, including Berrick [ 10 ] , open-source expressive faces [ 11 ] , and modular 3D-printed robotic heads [ 12 ] . However, many of these platforms focus mainly on mechanical expressiveness or platform design, without deeply integrating visual attention estimation, semantic scene interpretation, and state-based social regulation. Recent advances in vision–language models offer new possibilities for contextual interpretation in social robotics. Janssens et al. [ 15 ] highlight the potential of VLMs for multimodal social interaction, although their integration into real-time embodied systems remains challenging. In this context, VLMs are particularly relevant because they can provide semantic labels that go beyond geometric cues, such as distinguishing whether a user is attending to the robot, looking at a phone, or directing attention elsewhere. However, their lower update rate and higher inference latency make them less suitable as direct control signals in time-sensitive HRI loops. In this context, the present work contributes an applied case study that combines a fast geometric attention layer, an independent semantic VLM observer, and an expressive state machine in a physically accessible robotic head based on the InMoov ecosystem [ 13 ] . Rather than replacing geometric attention estimation with semantic interpretation, the proposed pipeline studies their complementary roles: the geometric layer provides high-frequency temporal control, while the VLM provides lower-frequency contextual interpretation for comparison and ambiguity analysis. 3 Methodology The proposed methodology evaluates a

Read the complete text