PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction
Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang · 2026
Read PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction by Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08125.txt.
Introduction
Lin and three colleagues present PolySLGen, an online framework for group speaking-listening replies. Human-like replies matter for natural group chats between humans and built AI. Current methods handle single modes or speaking-only replies in pairs, unfit for real social scenes, and miss silent cues and complex group dynamics key for engagement and chat flow. Given past talk and motion from all, it makes a future reply for a target person, including speech, body motion, and speaking state score.
Catalog record
| CREATOR | Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63544 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08125
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PolySLGen: Online Multimodal Speaking-Listening Reaction Generation in Polyadic Interaction Zhi-Yi Lin, Thomas Markhorst, Jouh Yeong Chew, Xucong Zhang Abstract Human-like multimodal reaction generation is essential for natural group interactions between humans and embodied AI. However, existing approaches are limited to single-modality or speaking-only responses in dyadic interactions, making them unsuitable for realistic social scenarios. Many also overlook nonverbal cues and complex dynamics of polyadic interactions, both critical for engagement and conversational coherence. In this work, we present PolySLGen , an online framework for Poly adic multimodal S peaking and L istening reaction Gen eration. Given past conversation and motion from all participants, PolySLGen generates a future speaking or listening reaction for a target participant, including speech, body motion, and speaking state score. To model group interactions effectively, we propose a pose fusion module and a social cue encoder that jointly aggregate motion and social signals from the group. Extensive experiments, along with quantitative and qualitative evaluations, show that PolySLGen produces contextually appropriate and temporally coherent multimodal reactions, outperforming several adapted and state-of-the-art baselines in motion quality, motion-speech alignment, speaking state prediction, and human-perceived realism. The source code and model are available at https://github.com/zylinzy/PolySLGen . 1 Introduction Figure 1 : Overview of the online reaction generation task in polyadic interaction. Given past group interactions, including speech and body motion of all participants, PolySLGen reasons about and generates the future reaction of the target participant (dark blue). The output includes speech, body motion, and a speaking state score (“Speak?”), which serves as an indicator of whether the generated reaction is speaking or listening. To support natural social interaction, embodied AI systems need to generate responses that coordinate speech and body motion with appropriate conversational turn-taking to signal attention, intention, and manage the flow of conversation Alarcon et al. 2021 ; Bisio et al. 2014 ; Urakami and Seaborn 2023 ; Russell and Harte 2025 . It is then important to model and generate both verbal and non-verbal behaviors to capture complex social dynamics of real social interactions for more efficient and expressive communication Al Moubayed et al. 2012 ; Trabelsi et al. 2019 ; Lee et al. 2024a ; Li et al. 2025 ; Rasenberg et al. 2022 ; Vogel et al. 2018 ; Vrij et al. 2019 . Multimodal Large Language Models (LLMs) have recently shown strong capabilities in motion understanding and generation Zhang et al. 2023a ; Ribeiro-Gomes et al. 2024 ; Zhou et al. 2024 , social cue interpretation Lee et al. 2024b , and multimodal question answering Zhan et al. 2024 ; Wu et al. 2024 ; Spitale et al. 2024 ; Liu et al. 2023 ; Alayrac et al. 2022 . Leveraging such models enables unified coordination of speech and body motion for more contextually consistent multimodal reactions Jiang et al. 2024 . Despite these advances, existing reaction generation approaches face several limitations. Photorealistic methods produce pixel-perfect outputs but lack the 3D physical grounding required for Embodied AI and Robotics Zhu et al. 2025 ; Siniukov et al. 2025 , where spatially consistent skeletal motions are needed for retargeting and control. Many approaches remain single-modal, generating one modality from another (e.g., motion-to-motion Siyao et al. 2024 , text-to-motion Ng et al. 2023b , and audio-to-motion Mughal et al. 2024 ; Ng et al. 2024 ). Most also rely on future context Siyao et al. 2024 ; Ghosh et al. 2024 ; Xu et al. 2024 ; Cen et al. 2025 , preventing online, causal generation from past observations Li et al. 2025 . Methods without speech generation Tanke et al. 2023b ; Peng et al. 2023 ; Jeong et al. 2024 ; Xiao et al. 2024 capture group dynamics only through body motion, limiting meaningful conversational participation. Recent work, SOLAMI Jiang et al. 2024 , explores multimodal LLMs for perceiving and generating motion and speech, but it focuses on speaking behavior and remains restricted to dyadic interactions. Turn-taking, the switch between speaking and listening, is essential for smooth and natural conversation, particularly in polyadic settings Żarkowski 2019 ; Ishii et al. 2022 . It ensures conversational coherence, maintains engagement, and allows each participant to respond appropriately to the actions of others. However, unified generation of both speaking and listening reactions remains underexplored, as prior works focus on either speaking Yi et al. 2023 ; Mughal et al. 2024 ; Ng et al. 2024 or listening responses Jonell et al. 2020 ; Ng et al. 2023b ; Ng et al. 2022 , limiting natural group interactions. Polyadic interactions are common in real-world scenarios such as group collaboration Ng et al. 2023a , education Voultsiou and Moussiades 2025 , and social support Leff et al. 2013 ; Kandalaft et al. 2013 , but introduce substantial complexity. As the number of participants increases, modeling their interdependent reactions, shaped by speech, gestures, and orientations, becomes more challenging Dahiya et al. 2023 ; Xu and LC 2022 ; Oertel et al. 2020 ; Wiemann and Knapp 2017 ; Al Moubayed et al. 2012 . Simply extending dyadic architectures to handle polyadic scenarios is computationally inefficient and often fails to capture higher-order dependencies. To address these challenges, we formulate online multimodal reaction generation as generating the future reaction of a single target participant in polyadic interactions. As shown in Fig. 1 , given past speech and motion from all participants, the model generates future speech, body motion, and a speaking state score that captures turn-taking behavior without enforcing hard transitions. This focus aligns with real-world scenarios, where embodied AI responds to others rather than predicting full group dynamics. We therefore propose PolySLGen, a novel framework that generates future reactions for a target participant from past group observations. A pre-trained LLM is adapted for conversation understanding, a pose fusion module aggregates past motions from all participants into compact embeddings, and a social cue encoder captures group-level attention toward the target participant, essential for realistic multi-party interaction. These modules handle variable group sizes and use fixed-length embeddings to preserve LLM context for larger groups. Combined with speech style generation and speaking state score prediction, PolySLGen produces coherent multimodal behavior and natural turn-taking. Extensive experiments and multi-aspect evaluation, including standard objective, social semantic, and human perception metrics, show that PolySLGen outperforms various baselines, remains robust to missing participants, and enables more realistic polyadic interactions. In summary, the main contributions of this paper include: • We are the first to propose an online multimodal reaction generation framework with both speaking and listening responses in polyadic interaction settings. • Our approach incorporates speaking state score prediction, allowing the system to dynamically alternate between speaking and listening responses. • Extensive experiments and multi-aspect evaluations show the proposed method outperforms variant baselines in motion, speech, and speaking state prediction. 2 Related Work Figure 2 : Overview of PolySLGen. Past multimodal group interactions are encoded into text \mathbf{e}^{\text{t}} , speech style \mathbf{e}^{\text{s}} , motion \mathbf{e}^{\text{m}} , and social cue \mathbf{e}^{\text{c}} embeddings as the inputs. The motion embeddings are aggregated by a pose fusion module, and the social cue encoder captures signals from non-target participants’ head orientations. The model generates target participant’s future reactions in text, speech style, and motion, through modality-specific decoders from embeddings \mathbf{h}^{\text{t}} , \mathbf{h}^{\text{s}} , and \mathbf{h}^{\text{m}} ,