Facial-Expression-Aware Prompting for Empathetic LLM Tutoring
Shuangquan Feng Laura Fleig Ruisen Tu Philip Chi Edmund Bu Melinda Ozel Junhua Ma Teng Fei Virginia R. de Sa Neurosciences Graduate Program, University of California San Diego, La Jolla, USA Department of Cognitive Science, University of California San Diego, La Jolla, USA Department of Computer Science and Engineering, University of California San Diego, La Jolla, USA Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, USA Department of Mathematics, University of California San Diego, La Jolla, USA Face the FACS Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, USA · 2026
Read Facial-Expression-Aware Prompting for Empathetic LLM Tutoring by Shuangquan Feng Laura Fleig Ruisen Tu Philip Chi Edmund Bu Melinda Ozel Junhua Ma Teng Fei Virginia R. de Sa Neurosciences Graduate Program, University of California San Diego, La Jolla, USA Department of Cognitive Science, University of California San Diego, La Jolla, USA Department of Computer Science and Engineering, University of California San Diego, La Jolla, USA Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, USA Department of Mathematics, University of California San Diego, La Jolla, USA Face the FACS Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15336.txt.
Introduction
Feng and eight colleagues test whether feeding facial-expression cues into tutoring prompts makes language-model tutors more empathetic. A simulated student shows varied facial behavior while four tutor variants, from text-only to Action Unit methods, hold 960 multi-turn chats across three model backbones. Action Unit conditioning improves empathetic response on every backbone, with peak-frame selection beating random frames, at no cost to clarity.
Catalog record
| CREATOR | Shuangquan Feng Laura Fleig Ruisen Tu Philip Chi Edmund Bu Melinda Ozel Junhua Ma Teng Fei Virginia R. de Sa Neurosciences Graduate Program, University of California San Diego, La Jolla, USA Department of Cognitive Science, University of California San Diego, La Jolla, USA Department of Computer Science and Engineering, University of California San Diego, La Jolla, USA Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, USA Department of Mathematics, University of California San Diego, La Jolla, USA Face the FACS Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, USA |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30525 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15336
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Facial-Expression-Aware Prompting for Empathetic LLM Tutoring Shuangquan Feng Laura Fleig Ruisen Tu Philip Chi Edmund Bu Melinda Ozel Junhua Ma Teng Fei Virginia R. de Sa Neurosciences Graduate Program, University of California San Diego, La Jolla, USA Department of Cognitive Science, University of California San Diego, La Jolla, USA Department of Computer Science and Engineering, University of California San Diego, La Jolla, USA Department of Electrical and Computer Engineering, University of California San Diego, La Jolla, USA Department of Mathematics, University of California San Diego, La Jolla, USA Face the FACS Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, USA Abstract Large language models (LLMs) enable increasingly capable tutoring-style conversational agents, yet effective tutoring requires sensitivity to learners’ affective and cognitive states beyond text alone. Facial expressions provide immediate and practical cues of confusion, frustration, or engagement, but remain underexplored in LLM-driven tutoring. We investigate whether facial-expression-aware signals can improve empathetic tutoring responses through prompt-level integration, without end-to-end retraining. We build a scalable simulated tutoring environment where a student agent exhibits diverse facial behaviors from a large unlabeled facial expression video dataset, and compare four tutor variants: a text-only LLM baseline, a multimodal baseline using a random facial frame, and two Action Unit estimation model (AUM)–based methods that either inject textual AU descriptions or select a peak-expression frame for visual grounding. Across 960 multi-turn conversations spanning three tutor backbones (GPT-5.1, Claude Ops 4.5, and Gemini 2.5 Pro), we evaluate targeted pairwise comparisons with five human raters and an exhaustive AI evaluator. AU-based conditioning consistently improves empathetic responsiveness to facial expressions across all tutor backbones, while AUM-guided peak-frame selection outperforms random-frame visual input. Textual AU abstraction and peak-frame visual injection show model-dependent advantages. Control analyses show that this improvement does not come at the expense of worse pedagogical clarity or responsiveness to textual cues. Finally, AI–human agreement is highest on facial-expression-grounded empathy, supporting scalable AI evaluation for this dimension. Overall, our results show that lightweight, structured facial expression representations can meaningfully enhance empathy in LLM-based tutoring systems with minimal overhead. Future work should evaluate these methods using real student facial data collected during authentic learning interactions, and examine whether improved empathetic responsiveness translates into measurable benefits in engagement and learning outcomes. I INTRODUCTION Large language models (LLMs) have enabled increasingly capable conversational systems in domains such as customer support and education [ 20 , 23 , 26 ] . However, effective human-aligned interaction often requires sensitivity to users’ affective and cognitive states, especially when users provide limited textual feedback [ 19 , 22 ] . Multimodal signals can provide complementary cues for such adaptation, and facial expressions are particularly practical indicators of states like confusion, frustration, or engagement [ 3 , 7 ] . Despite this, facial behavior remains underexplored in LLM-driven interaction frameworks compared to text or audio. AI tutoring offers a natural testbed for studying this gap: successful tutoring requires not only correct explanations but also timely, empathetic adjustments to learners’ states [ 25 , 6 ] . In this work, we investigate whether injecting facial-expression-aware signals into tutoring-style LLM responses can improve empathetic responsiveness without end-to-end retraining. We incorporate structured facial representations derived from Action Units (AUs) and compare two lightweight integration strategies: (i) converting AU estimates into concise textual descriptions appended to a text-only tutor prompt, and (ii) using AU-based saliency to select a peak frame for a multimodal tutor. Through controlled simulated multi-turn tutoring conversations and both human and AI evaluation, we analyze when and how facial-expression-awareness improves empathy in tutoring responses. II RELATED WORK Affect-Aware Tutoring Prior work in intelligent tutoring systems emphasizes that affective states such as confusion and frustration influence engagement and learning, motivating affect-aware interventions beyond purely cognitive optimization [ 25 , 6 ] . These findings suggest that tutoring systems should adapt not only to what learners say, but also to how they appear to be feeling during interaction. Facial Expression Representations and Multimodal LLMs Facial expression recognition (FER) methods commonly model either categorical emotions/valence-arousal [ 12 , 17 ] or anatomically grounded facial Action Units (AUs) from FACS [ 7 ] . AU-based representations are compositional and fine-grained, making them suitable for settings where learner states may not align with a small set of emotion labels [ 24 , 4 ] . In parallel, vision-language models align visual encoders with language backbones to enable multimodal reasoning [ 18 , 1 ] , with LLaVA as a representative instruction-tuned framework [ 13 ] . However, most general-purpose VLMs primarily capture object-level semantics and do not explicitly model facial affective cues. Our work bridges these directions by integrating AU-derived facial signals into an LLaVA-style tutoring framework via lightweight prompt-level conditioning, enabling targeted evaluation of facial-expression-aware empathy in tutoring interactions. III METHODS Fig. 1: Facial-expression-aware tutoring workflow and targeted comparisons. Given the same conversation history and a student’s facial expression video, an AU estimation model (AUM) enables two expression-aware integration strategies: (1) LLM+AUM converts estimated AU intensities into a textual description that is appended to the tutor prompt (AU \rightarrow Text); (2) MLLM+AUM selects a peak-expression frame as the visual input to a multimodal LLM tutor. Two baselines omit AUM: LLM uses text-only inputs, and MLLM uses a randomly sampled facial frame. Our analysis focuses on three pre-specified pairwise comparisons: (i) LLM+AUM vs. LLM, (ii) MLLM+AUM vs. MLLM, and (iii) LLM+AUM vs. MLLM+AUM. III-A Overview We study whether incorporating facial-expression-aware signals into an AI tutor’s input via prompt engineering can lead to more empathetic tutoring responses. Our core hypothesis is that explicitly modeling a student’s nonverbal emotional or cognitive state improves the tutor’s ability to generate empathetic responses, particularly when the student provides little or no textual feedback (see fig. 1 ). To test this hypothesis in a controlled and scalable manner, we construct a simulated multi-turn AI tutoring environment involving (i) a student agent equipped with facial expression videos, and (ii) multiple variants of an AI tutor that differ in how (or whether) they access and utilize facial expression information. We compare four tutor variants across multiple large language models (LLMs), using both human and AI-based evaluation focused on empathetic responsiveness. III-B Facial Expression Processing via AU Estimation To extract structured facial expression information from video, we use an Action Unit (AU) estimation model (AUM). Specifically, we use an IR50 backbone [ 5 ] pretrained on Glink360k [ 2 ] and fine-tuned on DISFA and DISFA+ [ 16 , 15 ] . Each video frame is preprocessed with face detection [ 14 ] , face alignment, and photometric normalization following [ 11 ] . The model estimates intensities for 12 AUs; in this work, we select the 8 most robust and informative AUs (AU1, AU2, AU4, AU5, AU9, AU12, AU15, AU17). The estimated AU intensities are translated into concise natural language descriptions (e.g., slightly raising eyebrows, moderately knitting eyebrows, strongly smiles), which can be directly incorporated into text-based prompts. III-C Facial Expression Video Source High-quality datasets capturing real user facial expressions during LLM interactions are scarce and expensive to collect. We therefore simulate realistic interaction videos by drawing from the HDFE-DevSplit-Unlabeled dataset, a development split of the Highly Diverse Facial Expressions (HDFE) dataset collected by the authors. It consists