User Experience Estimation in Human-Robot Interaction via Multi-Instance Learning of Multimodal Social Signals
Ryo Miyoshi, Yuki Okafuji, Takuya Iwamoto, Junya Nakanishi, Jun Baba · 2025
Read User Experience Estimation in Human-Robot Interaction via Multi-Instance Learning of Multimodal Social Signals by Ryo Miyoshi, Yuki Okafuji, Takuya Iwamoto, Junya Nakanishi, Jun Baba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23544.txt.
Introduction
Ryo Miyoshi and co-authors build in a 2025 method a way to judge how a person feels while using a social robot. Robots that adapt need such a read, yet past tools scored single facets like mood or engagement apart. Their data set joins face and voice signals from users over time.
A transformer model reads the signals in chunks, short and long, under a multi-instance scheme. That lets it catch how the experience shifts across a session, not just in a flash. In tests it beat human third-party judges at the same task. Finer reads of users could make robots respond with more care.
Catalog record
| CREATOR | Ryo Miyoshi, Yuki Okafuji, Takuya Iwamoto, Junya Nakanishi, Jun Baba |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23544
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
User Experience Estimation in Human-Robot Interaction via Multi-Instance Learning of Multimodal Social Signals Ryo Miyoshi, Yuki Okafuji, Takuya Iwamoto, Junya Nakanishi, Jun Baba Abstract In recent years, the demand for social robots has grown, requiring them to adapt their behaviors based on users’ states. Accurately assessing user experience (UX) in human-robot interaction (HRI) is crucial for achieving this adaptability. UX is a multi-faceted measure encompassing aspects such as sentiment and engagement, yet existing methods often focus on these individually. This study proposes a UX estimation method for HRI by leveraging multimodal social signals. We construct a UX dataset and develop a Transformer-based model that utilizes facial expressions and voice for estimation. Unlike conventional models that rely on momentary observations, our approach captures both short- and long-term interaction patterns using a multi-instance learning framework. This enables the model to capture temporal dynamics in UX, providing a more holistic representation. Experimental results demonstrate that our method outperforms third-party human evaluators in UX estimation. I Introduction The declining labor force and aging population have driven the increasing deployment of social robots in service industries, including customer support and product recommendations [ 1 , 2 , 3 , 4 , 5 ] . For these robots to be effective, they must understand users’ states and adjust their behavior accordingly [ 6 ] . Research has focused on estimating user states, primarily through externally observable behaviors. People often express their psychological states through nonverbal behaviors, which play a crucial role in interactions [ 7 ] . These nonverbal behaviors, collectively known as social signals [ 8 ] , include facial expressions, gaze, body posture, and vocal tone. In human-robot interaction (HRI), social signals have been widely utilized in two primary research areas: engagement assessment and sentiment analysis. Engagement is defined as the process of initiating, maintaining, and terminating an interaction, as well as the relational state between a user and a robot [ 9 ] . Many studies have quantified engagement as a continuous value based on observable behaviors such as gaze direction and posture [ 10 , 11 , 12 , 13 ] . While these engagement estimation models have been applied to robot behavior design, they primarily capture the level of interaction rather than the user’s subjective evaluation of the experience. Sentiment analysis, on the other hand, attempts to classify a user’s internal state as positive, neutral, or negative based on multimodal inputs [ 14 ] . This approach has been widely used in affective computing and HRI to design behaviors that promote positive emotional responses [ 15 , 16 , 17 , 18 , 19 ] . However, sentiment analysis provides only a simplified representation of user experience, making it challenging to capture the nuanced and multi-faceted aspects of interactions. In contrast, user experience (UX) is a broader and more comprehensive concept that encompasses various aspects of an interaction [ 20 ] . UX evaluations are commonly conducted using multi-dimensional scales, such as the User Experience Questionnaire (UEQ) [ 21 ] , which assesses aspects like attractiveness, efficiency, and novelty. While UX, engagement, and sentiment are distinct concepts, they are closely interrelated and influence one another. Engagement reflects the extent to which a user is involved in an interaction, while sentiment captures momentary emotional states. UX, in contrast, is a higher-level, multi-dimensional construct that integrates both engagement and sentiment, as well as other cognitive and affective responses. For example, a user may be highly engaged but still have a negative UX if the interaction feels frustrating. Conversely, positive sentiment alone does not necessarily translate to a satisfying UX if the interaction lacks efficiency or usefulness [ 22 ] . Since UX directly affects user satisfaction and the long-term adoption of robotic services, estimating UX in HRI is crucial for improving user-centered robot behaviors [ 23 , 24 ] . However, despite the growing interest in HRI, few studies have focused on automatically evaluating UX in human-robot interactions. Most existing research relies on post-interaction surveys, which are subjective and time-consuming, making large-scale analysis difficult. Moreover, while engagement and sentiment have been extensively studied in HRI, there is no established approach for systematically modeling and evaluating UX in an automated manner. To address this limitation, this study proposes an approach to automating UX evaluation in HRI by leveraging multimodal social signals. Specifically, we introduce a multi-instance learning framework that captures both short-term and long-term interaction patterns to assess UX. Unlike conventional engagement or sentiment models that are based on momentary observations, our method analyzes sequential patterns to construct a more comprehensive representation of UX. While large-scale vision-language models (VLMs) excel in many tasks, they struggle with video data, making it difficult to capture both short-term and long-term features in interactions. To overcome this, we evaluate our proposed method using data from human-robot interactions. The main contributions of this study are as follows: • We propose a Transformer-based multi-instance learning model for UX estimation in human-robot interaction, capturing both short- and long-term patterns. • We evaluate the proposed method by comparing its accuracy against multiple baselines, including third-party human evaluators and unimodal models. • We analyze the effectiveness of facial expressions and voice in UX estimation, demonstrating the benefits of a multimodal approach. II Related Work II-A Assessment of Engagement Engagement is defined as the process of initiating, maintaining, and terminating an interaction, as well as the relational state between a user and an entity [ 9 ] . In HRI, engagement is often measured as the degree to which a user is involved in an interaction with a robot. Many studies have proposed methods to estimate and enhance engagement by analyzing observable behavioral cues such as gaze direction, body posture, and facial expressions [ 10 , 11 , 12 , 13 ] . Engagement is typically quantified as a continuous value ranging from 0 to 1, representing an externally observable state in HRI. For instance, Salvatore et al. [ 10 ] estimated engagement using user gaze and gestures. Rudovic et al. [ 11 ] developed a deep-learning-based approach for engagement estimation in child–robot interactions. Del et al. [ 12 ] introduced an engagement assessment method using first-person robot vision, while Del et al. [ 13 ] applied reinforcement learning to maximize engagement in tour-guide robots. Although engagement estimation provides valuable insights into user-robot interactions, it has inherent limitations. Engagement is an externally observable state rather than a direct measure of user perception. Annotations are typically provided by third-party observers, which may not necessarily align with the user’s subjective evaluation [ 25 ] . For instance, a user who maintains eye contact with a robot might be classified as highly engaged, even if they feel uncomfortable or frustrated. This suggests that engagement alone is insufficient to fully capture the overall quality of an interaction. II-B Sentiment Analysis Sentiment analysis estimates a user’s psychological state by classifying their emotional response into discrete categories such as positive, neutral, or negative [ 14 ] . This method has been widely utilized in affective computing and HRI for developing adaptive robot behaviors [ 15 , 16 , 17 , 18 , 19 ] . Several studies have explored multimodal sentiment analysis. Tomimasu et al. [ 16 ] proposed an