Classification of User Satisfaction in HRI with Social Signals in the Wild

Michael Schiffmann, Sabina Jeschke, Anja Richert · 2025

Read Classification of User Satisfaction in HRI with Social Signals in the Wild by Michael Schiffmann, Sabina Jeschke, Anja Richert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03945.txt.

Introduction

Knowing whether a museum visitor enjoyed talking to a robot is normally gathered by handing them a questionnaire afterwards. Michael Schiffmann, Sabina Jeschke, and Anja Richert test an automatic alternative in a 2025 human-computer interaction preprint. Their data comes from a field trial at Bonn's Deutsches Museum, where a Furhat robot head served as an information point for real visitors.

The trial recorded dozens of single-user interactions, each with questionnaire answers and video. From the video, the team extracted time series of body pose, facial expression, and the visitor's distance from the robot. Classifiers trained on those social signals, under three different feature designs, learned to flag interactions with low satisfaction. The method worked without any hand-labelled training data. The authors see automated satisfaction sensing as a feedback channel that could steadily improve social robots and the experiences they offer.

Catalog record

CREATORMichael Schiffmann, Sabina Jeschke, Anja Richert
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03945

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Classification of User Satisfaction in HRI with Social Signals in the Wild Michael Schiffmann, Sabina Jeschke, Anja Richert Abstract Socially interactive agents (SIAs) are being used in various scenarios and are nearing productive deployment. Evaluating user satisfaction with SIAs’ performance is a key factor in designing the interaction between the user and SIA. Currently, subjective user satisfaction is primarily assessed manually through questionnaires or indirectly via system metrics. This study examines the automatic classification of user satisfaction through analysis of social signals, aiming to enhance both manual and autonomous evaluation methods for SIAs. During a field trial at the Deutsches Museum Bonn, a Furhat Robotics head was employed as a service and information hub, collecting an "in-the-wild" dataset. This dataset comprises 46 single-user interactions, including questionnaire responses and video data. Our method focuses on automatically classifying user satisfaction based on time series classification. We use time series of social signal metrics derived from the body pose, time series of facial expressions, and physical distance. This study compares three feature engineering approaches on different machine learning models. The results confirm the method’s effectiveness in reliably identifying interactions with low user satisfaction without the need for manually annotated datasets. This approach offers significant potential for enhancing SIA performance and user experience through automated feedback mechanisms. 1 Introduction Socially interactive agents (SIA) are applied in various scenarios, such as in public spaces like a museum [ 6 , 28 , 32 ] . SIAs can be viewed from a technological and application-specific perspective: On the one hand, it is crucial by definition that they continuously adapt to users. On the other hand, operators and researchers are interested in whether SIAs deliver the desired performance in productive use. For both approaches, the automatic measurement of user satisfaction can be crucial for the evaluation and improvement of SIAs. The integration of UX data is an essential development factor [ 26 , 4 ] , particularly in service and information environments where the goal is to assist users. There are various ways to evaluate SIAs, for instance, indirectly through objective system metrics regarding the efficiency of the SIA’s speech dialogue system, such as the number of turns or dialogue duration [ 7 , 29 ] . Alternatively, metrics like the number of different dialogue paths can be used to assess the complexity of potential dialogues, which, according to [ 7 ] , is an indicator of efficiency and satisfaction. Direct evaluation of subjective user satisfaction is only possible through questionnaires or interviews. An approach that captures this implicitly is necessary for the automatic collection of user satisfaction since active feedback requests interrupt the interaction, and post-interaction feedback requests risk low participation. A video-based possibility involves capturing social signals that allow insights into the user’s internal state. Nonverbal social signals or cues can be used to make these inferences [ 20 ] and like [ 31 , 24 , 15 ] showed that annotated video data can be used to classify quality features like user satisfaction or user experience. Assessing the user’s state is not only of interest for adaptation but also for evaluating the SIA’s performance in specific applications. In this paper, we explore how individual user satisfaction ratings can be automatically classified using social signals as a source for machine learning feature extraction, specifically body language, facial expressions, and distance, to predict user satisfaction. We use an in-the-wild dataset consisting of single user interactions and corresponding post user satisfaction ratings of each particular user. To achieve this, we utilize data collected from a field study with an autonomous SIA deployed as a service and information hub at the Deutsches Museum Bonn. Our method distinguishes itself from the state-of-the-art by using an holistic approach to capture social signals from body language, not requiring additional annotations from third parties or users. We compare three feature engineering methods to demonstrate how to classify interactions with low user satisfaction and interactions with medium to high user satisfaction ratings. Despite unpredictable conditions, realistic interaction data offer valuable insights, allowing for the study of unbiased interactions with an autonomous SIA. In the following, we first discuss the psychological background of social signals and related work. We then present the implemented system and experimental setting. In the methods section, we explain data collection, feature engineering, and the composition of the classification task before we finally present the results and discuss them in terms of the peculiarities of the "in-the-wild" approach and further research avenues arising from this work. 2 Related Work In social interactions, individuals exchange information through verbal and non-verbal communication with their conversation partners. Social signals, including body language, gestures, and facial expressions, serve various purposes in this context and help convey one’s emotional state or stance towards something (social attitude) [ 21 ] . In HRI, social signals have been effectively utilized to evaluate various psychological constructs, such as rapport [ 20 , 30 ] , user satisfaction [ 31 , 24 ] , user experience [ 15 ] , user engagement [ 15 ] , and affective assignment [ 19 , 13 ] within Russell’s circumplex model of affect (two-dimensional model for emotions) [ 22 ] . Müller et al. [ 20 ] predicted perceived low rapport versus medium/high rapport in human-to-human group interactions using automatically derived features from social signals. They found that facial cues, including a variety of features such as indicators of happiness, synchrony indicators, the amount of mutual facing, among others were the strongest predictors of low rapport, achieving an average accuracy of 70%. Jokinen and Wilcock [ 15 ] showed in a laboratory HRI Experiment that certain aspects of user experience have a significant correlation with the user behavior and that it is possible to predict the user’s experience overall and its aspects of usability, expressiveness, responsiveness, and interface from a manually annotated dataset. Wei et al. [ 31 ] studied user satisfaction (well-coordinated, awkward, unfriendly) in a Wizard-of-Oz interaction with a digital agent, assessing it through user questionnaires and self-assessments. They successfully used deep learning on multi-modal verbal and non-verbal social signals, finding that multi-modal models outperformed uni-modal ones in predicting user satisfaction. Schiffmann et al. [ 24 ] showed that user satisfaction can be predicted based on annotated valence and arousal time series features and the user satisfaction value of the respective users of human-robot interaction which was recorded in the wild. Using a generated skeleton model, McColl et al. [ 19 ] demonstrated how to predict valence and arousal levels by translating the user’s body language into angles, velocities, and expansion. This approach was subsequently utilized with a rater-based method for annotating interaction data, resulting in the development of separate valence and arousal models for training and prediction. However, all these approaches share the commonality of being based on controlled laboratory studies, with training data generated through an annotation process, except in the case of [ 20 , 24 ] . 3 System and Setting The field trial took place from July 23, 2024, to August 15, 2024, at the Deutsches Museum Bonn (DMB). The DMB is presenting an exhibition on artificial intelligence with interactive exhibits to bring artificial intelligence closer to a broad audience. The robot "Mira", developed to

Read the complete text