Xpress: A System For Dynamic, Context-Aware Robot Facial Expressions using Language Models

Victor Nikhil Antony, Maia Stiber, Chien-Ming Huang · 2025

Read Xpress: A System For Dynamic, Context-Aware Robot Facial Expressions using Language Models by Victor Nikhil Antony, Maia Stiber, Chien-Ming Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00283.txt.

Introduction

Victor Antony, Maia Stiber, and Chien-Ming Huang give a system that makes robot faces shift with talk sense by use of language models, in a 2025 paper. Face moves shape trust and warmth in human robot chats. Past face sets are hand made, stiff across scenes, and thin in range, which dulls long chats.

Their three step way codes time flow, weighs scene sense, and writes face code live. Two lab studies and one kid and kin case test the tool. Faces came out apt, rich, and quick to shift with scene turns. One tool thus fits tales and chats on mixed bots.

Catalog record

CREATORVictor Nikhil Antony, Maia Stiber, Chien-Ming Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-00283

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Xpress: A System For Dynamic, Context-Aware Robot Facial Expressions using Language Models Thanks: CRediT author statement . Conceptualization, Methodology, Writing-Review&Editing, Visualization (all authors) Formal Analysis, Writing-Original Draft (VNA, MS), Software, Validation, Investigation, Data Curation, Project Administration (VNA) Supervision, Funding (CMH). Thanks: AI Use. Text edited with LLM; output checked for correctness by authors. Victor Nikhil Antony, Maia Stiber, Chien-Ming Huang Abstract Facial expressions are vital in human communication and significantly influence outcomes in human-robot interaction (HRI), such as likeability, trust, and companionship. However, current methods for generating robotic facial expressions are often labor-intensive, lack adaptability across contexts and platforms, and have limited expressive ranges—leading to repetitive behaviors that reduce interaction quality, particularly in long-term scenarios. We introduce Xpress , a system that leverages language models (LMs) to dynamically generate context-aware facial expressions for robots through a three-phase process: encoding temporal flow, conditioning expressions on context, and generating facial expression code. We demonstrated Xpress as a proof-of-concept through two user studies ( n = 15 × 2 n=15\times 2 ) and a case study with children and parents ( n = 13 n=13 ), in storytelling and conversational scenarios to assess the system’s context-awareness, expressiveness, and dynamism. Results demonstrate Xpress’s ability to dynamically produce expressive and contextually appropriate facial expressions, highlighting its versatility and potential in HRI applications. Fig. 1: We present Xpress, a system that dynamically generates expressive, context-aware facial expressions for robots. We demonstrate its ability to generate facial expressions in two distinct contexts: storytelling and conversation. I Introduction a Facial expressions are a fundamental aspect of human communication, playing a crucial role in conveying emotions and intentions [ 1 , 2 , 3 ] . Similarly, robotic facial expressions are essential not only for expressing emotions and intentions but also for signaling functional and behavioral traits such as intelligence, agency, and personality [ 4 , 5 , 6 ] . These expressions significantly influence human-robot interaction outcomes, affecting factors such as likeability, companionship, trustworthiness, and collaboration [ 7 , 8 , 9 , 10 ] , which are crucial for long-term engagement. Conversely, incongruent robot expressions can negatively impact interactions [ 11 , 12 , 13 ] , underscoring the need for context-appropriate facial expressions. Two primary approaches exist for producing robot facial expressions: automated methods, which leverage machine learning but require extensive training data and often encounter issues with transferability across different contexts and robots [ 14 ] ; and handcrafting methods [ 15 ] , which are time-consuming and tend to yield repetitive behaviors due to limited emotional diversity. In long-term, repeated interactions, the limited expressive range of existing methods leads to predictable and repetitive behaviors that negatively influence the interaction and relationship [ 16 , 17 ] . To address the limitations of current robot facial expression generation approaches, we introduce Xpress , a system that leverages language models to process contextual information and dynamically generate robot facial expressions (see Fig. Xpress: A System For Dynamic, Context-Aware Robot Facial Expressions using Language Models ). Language models (LMs) demonstrate ability to recognize socio-emotional contexts in text inputs [ 18 , 19 , 20 , 21 , 22 ] and to generate code [ 23 , 24 , 25 , 26 ] . The core innovation of Xpress is incorporating LMs in a three-phase process to analyze interaction content ( i.e., robot dialogue, user speech), socio-emotional context, and generation history—enabling dynamic production of contextually appropriate expressions that are expressive. We validated Xpress as a proof-of-concept through two user studies ( n=15\times 2 ) and a case study ( n=13 ) across two domains. The first user study involved a storytelling robot to test the system’s ability to adapt expressions to complex narrative contexts; a case study with parents and children was also conducted to assess Xpress’s validity within this domain. The second focused on a conversational robot to evaluate Xpress’s performance in real-time, fluid exchanges. For each user study, we evaluated the system along three key dimensions: how context-aware the facial expressions were, how expressive they were, how dynamic the generations were. This work makes three key contributions: 1. Design, development, and open-sourcing of Xpress . 2. Demonstration of Xpress in two domains— storytelling and conversational interaction —as a proof-of-concept. 3. Insights from our exploration with Xpress , highlighting the potential and challenges of using state-of-the-art LMs for generating facial expressions in robotic systems. II Background and Related Work II-A Emotions and Human-Robot Interaction Robots that express emotions enhance social presence and user engagement, making interactions more meaningful [ 27 , 5 , 28 ] . Emotion expression improves perceptions of likeability, companionship, trust, intelligence, and collaboration [ 29 , 4 , 7 , 8 , 9 , 10 ] , and fosters human empathy [ 30 ] . In therapy [ 31 ] and education [ 32 ] , emotion expression enhances patient experiences and motivates students, demonstrating their practical value. However, robots must express emotions appropriately, as mismatched responses can harm interactions [ 11 , 12 ] . To ensure accuracy, current emotion recognition methods rely on cues such as facial expressions and body language [ 33 , 34 ] assuming strong correlations between physical expressions, subjective feelings, and social meaning. These methods often rely on discretizing emotions into preset categories [ 35 ] , which can fail to capture the nuances of complex interactions. Emotions in social contexts are rarely discrete, making rigid categorization limiting [ 36 , 37 , 38 ] . Advancements in LMs have shown state-of-the-art potential in recognizing socio-emotional context across various domains [ 19 , 21 ] ; LMs can interpret nuanced emotion in interviews with children [ 22 ] , dialogic interactions [ 20 ] , and speech data [ 18 ] . Xpress leverages this capability to understand socio-emotional context to capture different contextual variables to inform the expression generation. II-B Generation of Facial Expressions for Robots Facial expressions are a common method for affect communication in robots [ 15 , 39 ] . Generating appropriate facial expressions has been a focus of research, with methods ranging from handcrafted designs [ 15 , 40 ] —which are labor-intensive and limited in expressiveness—to automated methods using pre-defined scripts [ 41 ] or machine learning techniques ( e.g., k-nearest neighbors [ 42 ] and generative adversarial networks [ 43 ] ). These automated methods often require large datasets and lack transferability across robotic systems, especially those with varying levels of anthropomorphism. Language models have demonstrated state-of-the-art capabilities in generating programs for robots across different platforms and tasks, such as manipulation and planning [ 23 , 24 , 25 , 26 ] . Recent work has also utilized LMs to generate individual non-verbal behaviors for robots [ 44 ] . Xpress integrates LM code generation capability and pushes the state-of-the-art to generate robotic facial behaviors across complex interactions. III Xpress: Generating Dynamic, Context-Aware Facial Expressions System Overview. Xpress follows a three-phase process to generate context-aware facial expressions as a temporal trajectory, denoted as \tau_{\text{face}} , which is conditioned on both the interaction content ( e.g., robot speech, user speech) and socio-emotional context ( e.g., tone, interaction background). Fig. Xpress: A System For Dynamic, Context-Aware Robot Facial Expressions using Language Models

Read the complete text