Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing HCI theory, concepts and models
Tongfei Bian, Mathieu Chollet, Tanaya Guha · 2026
Read Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing HCI theory, concepts and models by Tongfei Bian, Mathieu Chollet, Tanaya Guha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10895.txt.
Introduction
Bian, Chollet, and Guha propose SocialLDG, a multi-task frame teaching robots to read social interaction by modeling the live link between users' hidden states and visible actions. Both spring from one socio-cognitive process and shape each other over time. A language model supplies word priors for six tasks, while dynamic graph learning tracks how task links shift over time. SocialLDG tops prior results on two public human-robot sets, learns new tasks smoothly without catastrophic forgetting, and reveals how interactions unfold and how states and actions steer choice.
Catalog record
| CREATOR | Tongfei Bian, Mathieu Chollet, Tanaya Guha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40405 |
| LICENSE | http://creativecommons.org/publicdomain/zero/1.0/ |
| RIGHTS | CC0 |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10895
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Teaching Robots to Interpret Social Interactions through Lexically-guided Dynamic Graph Learning CCS: Human-centered computing Human computer interaction (HCI) CCS: Human-centered computing HCI theory, concepts and models Tongfei Bian, Mathieu Chollet, Tanaya Guha Abstract . For a robot to be called socially intelligent, it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately. In this work, we investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user’s internal states (latent) and actions (observable state). Our premise is that these states arise from the same underlying socio-cognitive process and influence each other dynamically. Drawing inspiration from theories in Cognitive Science, we propose a novel multi-task learning framework, termed as SocialLDG that explicitly models the dynamic relationship among the states represent as six distinct tasks. Our framework uses a language model to introduce lexical priors for each task and employs dynamic graph learning to model task affinity evolving with time. SocialLDG has three advantages: First, it achieves state-of-the-art performance on two challenging human-robot social interaction datasets available publicly. Second, it supports strong task scalability by learning new tasks seamlessly without catastrophic forgetting. Finally, benefiting from explicit modelling task affinity, it offers insights on how different interactions unfolds in time and how the internal states and observable actions influence each other in human decision making. Our code is available github.com/biantongfei/SocialLDG . 1. Introduction Figure 1 . Dynamic reasoning in social HRI multitasking. Social robots are now increasingly more common in our social spaces. Achieving smooth human-robot interaction (HRI) in these social settings requires rethinking how robots interact and engage with humans moving beyond physical collaboration ( 6 ) . For a robot to be called socially intelligent , it must be able to infer users internal states from their current behaviour, predict the users future behaviour, and if required, respond appropriately ( 19 ) . In this work, we investigate how robots can be endowed with such social intelligence by modelling the dynamic relationship between user’s internal states (latent) and actions (observable) . The majority of social robots are reactive ( 39 ; 11 ) , where they wait for the user to start an interaction. In contrast, we consider a paradigm where robots are proactive i.e., they are allowed to start an interaction ( 5 ; 1 ) . Our task therefore starts with identifying (among those in the robot’s field of view) the potential user(s) i.e, to infer the intent of a human to interact. Intent is an internal state (latent) that can only be estimated from observable social cues, such as closeness to the robot (see Fig. 1 ). We also consider inferring user’s internal state of attitude towards the robot, which influences observable actions . Our premise is that intent-attitude-actions are not isolated components, rather part of the same underlying socio-cognitive process, and therefore, should be predicted jointly. While it is clear that these perceptual tasks are related, there is no widely accepted cognitive or psychological theory that describes their relationship. Past works ( 5 ; 4 ; 28 ; 45 ) assumed static relationship among similar perceptual tasks, which is inadequate for capturing the complex, evolving relationship among the latent internal states and observable states i.e, actions. In this work, we thus adopt a completely data-driven approach and learn the relationship using learnable dynamic graphs within a Multitask Learning (MTL) framework. Our dynamic graph approach enables modelling the variability and richness of inter-task relationship depending on situations (e.g., uninterested vs enthusiastic user) and stages of interaction (e.g., early vs late stage of interaction when attitude become apparent). To this end, we propose SocialLDG - a novel MTL framework for a robot to understand social interactions by explicitly modelling the dynamic relationship among user’s latent internal states (intent, attitude) and observable states (actions) via dynamic graphs. SocialLDG is a novel MTL framework for understanding human-robot social interactions. The input to the SocialLDG is a set of non-verbal embeddings, which are extracted from AlphaPose ( 9 ) encodings of the face, hands, and full body using a pretrained Autoencoder. Inspired by the Parallel Constraint Satisfaction (PCS) theory in cognitive psychology ( 20 ) , SocialLDG models decision-making as a network where different constraints (tasks) i.e., latent internal states and observable actions in our case interact simultaneously. This is modelled using a graph of six nodes, each representing a task (constraint in PCS theory), where the edges indicate the direction and strength of inter-task interaction. The six tasks span physical and cognitive aspects: immediate physical states (current contact and current action), anticipatory physical states (future contact and future action), and internal state inference (intent and attitude). These graphs are learnable over time (Task Affinity matrix) for every scenario, allowing us to learn the relationship among the latents and observables automatically from data. The node features are initialised using a frozen language model to provide a strong prior for the tasks. A graph transformer guided by semantic edge biases ( 42 ) aggregates and makes predictions. To summarise, our contributions are as follows: 1 A lexically-guided dynamic graph MTL framework: This is developed by incorporating task-specific context through a language model that acts as a prior for each task. Our framework, SocialLDG, learns the temporal inter-task relationship through a series of dynamic graphs, thereby offering interpretability and improving the overall performance. 2 Task scalability: We demonstrate that SocialLDG can rapidly learn new tasks without catastrophically forgetting previously learned initial tasks. This is attributed to its explicit modeling of inter-task relationship and general encoder-based social signal representations. 3 Interpretable task relationship: Our framework explicitly models the dynamics of inter-task relationship using a task affinity matrix ; this captures how different social perception evolves with time and provides insights about how latent and observable states inform each other during decision-making. 2. Related Work Social interaction modelling. Current literature on social interaction understanding range from recognizing low-level actions to inferring higher-level constructs such as intent prediction. Studies have explored recognising social actions (e.g., waving and approaching from a robot’s perspective ( 34 ; 43 ) and also, predicting future actions ( 33 ) . Other works have predicted user engagement during an gaze-based interaction and distance to the robot ( 17 ) . Studies that focus on understanding users’ internal states involve integrating facial expressions, audio signals and conversion scripts to recognise user emotions during HRI ( 27 ; 21 ; 14 ) . Users interaction intent has been predicted for using cues related to body movement in another recent work ( 1 ) . All these works treat components of human interaction (e.g., action vs emotion) as separate entities and ignore their inherent coupling. A few recent studies ( 5 ; 4 ) have attempted to jointly model social interaction as a holistic process where internal states and observables states are simultaneously predicted. However, these attempts assume fixed relationships among the various tasks, and fail to capture how the tasks mutually influence the social perception and decision making. Multitask learning (MTL). MTL enhances generalisation and efficiency through representation sharing across related tasks. Earlier approaches have relied on hard parameter sharing ( 24 ) , where