Multi-Party Conversational Agents: A Survey

Sagar Sapkota, Mohammad Saqib Hasan, Mubarak Shah, Santu Karmaker · 2025

Read Multi-Party Conversational Agents: A Survey by Sagar Sapkota, Mohammad Saqib Hasan, Mubarak Shah, Santu Karmaker (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18845.txt.

Introduction

Sagar Sapkota, Mohammad Saqib Hasan, Mubarak Shah, and Santu Karmaker survey chatbots built for group conversation. Talking with several people at once is far harder than a two-person chat. The agent must track who means what, and who knows what. Social dynamics ride on top of the words themselves.

The survey sorts the field by three questions. Can the agent model each speaker's state of mind? Can it grasp the content of the talk? Can it foresee where the talk goes and act well? Methods range from classic machine learning to large language models and multi-sense systems. Modeling minds, the authors stress, is the core skill still missing. Systems that read images and voice as well as text look promising but remain little explored.

Catalog record

CREATORSagar Sapkota, Mohammad Saqib Hasan, Mubarak Shah, Santu Karmaker
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18845

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Party Conversational Agents: A Survey Sagar Sapkota, Mohammad Saqib Hasan, Mubarak Shah, Santu Karmaker Abstract Multi-party Conversational Agents (MPCAs) are systems designed to engage in dialogue with more than two participants simultaneously. Unlike traditional two-party agents, designing MPCAs faces additional challenges due to the need to interpret both utterance semantics and social dynamics. This survey explores recent progress in MPCAs by addressing three key questions: 1) Can agents model each participants’ mental states? ( State of Mind Modeling ); 2) Can they properly understand the dialogue content? ( Semantic Understanding ); and 3) Can they reason about and predict future conversation flow? ( Agent Action Modeling ). We review methods ranging from classical machine learning to Large Language Models (LLMs) and multi-modal systems. Our analysis underscores Theory of Mind (ToM) as essential for building intelligent MPCAs and highlights multi-modal understanding as a promising yet underexplored direction. Finally, this survey offers guidance to future researchers on developing more capable MPCAs. 1 Introduction Figure 1: Example of a multi-party conversation demonstrating key challenges that MPCAs must handle. At each time step of the conversation, the MPCA must identify the states of mind of each participant (e.g., curiosity, frustration, etc.), have semantic understanding of the conversation (e.g., speaker actions like criticize and explain, dialog summary, etc.), and be able to take the appropriate action (e.g., response, turn-taking, identifying addressee, etc.). Combining these capabilities makes for a social and intelligent agent. Developing agents for conversational understanding and natural dialogue generation has been a long-standing research agenda within the NLP community Wang et al. 2024a . Traditionally, most conversational agents have been designed for two-party interactions Zheng et al. 2022 . However, real-world conversations—such as group chats, team meetings, online gaming, and customer support forums—often involve more than two participants. As a result, there has been growing interest in developing agents capable of engaging with multiple participants simultaneously, commonly referred to as Multi-Party Conversational Agents (MPCAs). Compared to traditional two-party systems, building MPCAs presents greater challenges due to the complex dynamics in multi-party interactions. Effective MPCAs must simultaneously infer participants’ mental states Cohen et al. 2024 , comprehend user content Ma et al. 2023a , and anticipate future dialogue flow Houde et al. 2025 . Mastering these capabilities is crucial for enabling collaborative tools, socially adept robots, and intelligent assistants. For instance, Figure 1 illustrates a two-speaker interaction with an agent, highlighting core challenges such as emotional variance, evolving latent states, and context-sensitive response planning Wei et al. 2023 ; Houde et al. 2025 . While recent advances in conversational AI AI 2024 ; OpenAI 2024 ; Qwen et al. 2025 ; Jiang et al. 2024a have improved these components individually, integrating them into fully functional MPCAs still remains an open challenge Zhao et al. 2023 ; Tan et al. 2023 . This research gap has recently attracted significant attention from the community, leading to a surge in efforts to design multi-party conversational systems Ganesh et al. 2023 . In synergy with with recent advances in generative AI, particularly Large Language Models (LLMs), most of these efforts have focused on leveraging LLMs for this domain Wang et al. 2024c ; Tan et al. 2023 . As such, this survey aims to serve as a comprehensive resource for AI researchers interested in developing MPCAs. We review over 70 research articles relevant to multi-party conversations and categorize their contributions into three broad themes: State of Mind Modeling: Works under this theme study the following question: “ Can models accurately infer each participant’s mental and emotional states? ” Key tasks under this theme include emotion recognition, participants’ engagement detection, personality identification, and recognition of user intents. Semantic Understanding : Under this theme, works assess MPCAs’ ability to properly understand utterances and the context of the multi-party dialogue. This covers tasks like dialogue summarization, conversation disentanglement, discourse structure analysis, and representation learning. Agent Action Modeling: Tasks pertaining to this theme describe MPCAs’ ability to predict future conversation flow. This involves tasks like turn detection, addressee selection, and response selection/generation. For each of the aforementioned themes, we start off by discussing research works utilizing traditional ML algorithms Bayser et al. 2019 ; Majumder et al. 2019 ; Ghosal et al. 2019 and NLP models Shen et al. 2020 ; Wen et al. 2024 ; Li et al. 2021b . We then move towards Large Language Models (LLMs) as many MPC tasks are being modeled by leveraging LLMs either through fine-tuning or prompting Wu et al. 2024 ; Li et al. 2025 ; Jiang et al. 2024b . Finally, since the proliferation of multi-modal data (e.g., meeting videos, group chats with audio-visual context) Zhang et al. 2024a ; Carletta et al. 2005 ; Sasu et al. 2025 ; Müller et al. 2024 , we discuss works where MPC tasks have been studied using visual and auditory cues Yoon et al. 2024 ; Lee and Deng 2024 ; Jain et al. 2023 . In addition, we discuss the major datasets associated with these tasks. Based on our thematic analysis of MPCA research, we identify two promising future directions: integrating Theory of Mind (ToM) Premack and Woodruff 1978 into multi-party dialogue systems and advancing Multi-modal Fusion and Grounding . For ToM integration, we highlight gaps in the current literature and suggest tailoring models to infer mental states within task-specific contexts. For multi-modal grounding, we highlight that key tasks—such as conversation disentanglement, representation learning, addressee selection, and response generation—remain largely underexplored in multi-modal settings. Consequently, even state-of-the-art multi-modal MPCAs often struggle to simulate realistic multi-party interactions. We then discuss how incorporating open-ended multi-modal benchmarks Tang et al. 2025 can enhance MPCA development. We hope that the insights drawn from our analysis will guide the community for future research toward more robust and socially aware MPCA systems. 2 A Thematic Taxonomy {forest} Figure 2: Thematic taxonomy of MPC tasks and recent works focusing on these tasks. To organize this survey paper, we create a taxonomy of existing research efforts on multi-party conversations (MPC) by categorizing them into three core themes: State of Mind Modeling , Semantic Understanding , and Agent Action Modeling . This taxonomy reflects essential capabilities for human-like social communication in group settings. Indeed, a socially intelligent agent must determine when to speak, whom to address, and what to say, collectively referred to as actions . According to Social Identity Theory Tajfel and Turner 1979 , such actions are influenced by the agent’s own and others’ mental states, while reciprocal effects also exist, where actions can shape mental states Hatfield et al. 1993 . Semantic understanding Kopp and Krämer 2021 further mediates these dynamics by enabling agents to interpret social cues and context Rijpma et al. 2023 . As illustrated in Figure 1 , these three components interact dynamically throughout the course of a conversation. Next, we formalize our proposed taxonomy by mapping existing MPC tasks to these three themes as follows: tasks aligned with State of Mind Modeling include emotion recognition, personality detection, dialog act recognition, and engagement detection; tasks supporting Semantic Understanding include dialog summarization, conversation disentanglement, discourse analysis, and representation learning; and tasks contributing to Agent Action Modeling include turn detection, addressee selection, and response generation. A visual taxonomy of

Read the complete text