Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Games Journal: Frontiers in Artificial Intelligence

Ryota Nonomura, Hiroki Mori · 2024

Read Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Games Journal: Frontiers in Artificial Intelligence by Ryota Nonomura, Hiroki Mori (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04937.txt.

Introduction

Ryota Nonomura and Hiroki Mori build, in twenty twenty-four, a framework that lets several AI agents hold a natural group discussion in a murder mystery game. Multi-agent language model systems show promise, but smooth dialogue control among agents is still hard. They borrow rules from conversation analysis: adjacency pairs, where one utterance type calls for a matching reply, and turn-taking norms. Their framework, Murder Mystery Agents, applies those rules to agent dialogue. Next speaker choice uses adjacency pairs plus a self-selection step that weighs each agent's inner state.

The murder mystery game demands social reasoning and bargaining over fragmentary clues. The team analyzed utterances that led to dialogue breakdowns. It also ran automatic and human evaluations. Next speaker selection cut breakdowns sharply and improved information sharing.

Catalog record

CREATORRyota Nonomura, Hiroki Mori
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of Turn-taking in Murder Mystery Games Journal: Frontiers in Artificial Intelligence Ryota Nonomura, Hiroki Mori Abstract Multi-agent systems utilizing large language models (LLMs) have shown great promise in achieving natural dialogue. However, smooth dialogue control and autonomous decision making among agents still remain challenges. In this study, we focus on conversational norms such as adjacency pairs and turn-taking found in conversation analysis and propose a new framework called “Murder Mystery Agents” that applies these norms to AI agents’ dialogue control. As an evaluation target, we employed the “Murder Mystery” game, a reasoning-type table-top role-playing game that requires complex social reasoning and information manipulation. In this game, players need to unravel the truth of the case based on fragmentary information through cooperation and bargaining. The proposed framework integrates next speaker selection based on adjacency pairs and a self-selection mechanism that takes agents’ internal states into account to achieve more natural and strategic dialogue. To verify the effectiveness of this new approach, we analyzed utterances that led to dialogue breakdowns and conducted automatic evaluation using LLMs, as well as human evaluation using evaluation criteria developed for the Murder Mystery game. Experimental results showed that the implementation of the next speaker selection mechanism significantly reduced dialogue breakdowns and improved the ability of agents to share information and perform logical reasoning. The results of this study demonstrate that the systematics of turn-taking in human conversation are also effective in controlling dialogue among AI agents, and provide design guidelines for more advanced multi-agent dialogue systems. 1 Introduction The emergence of large language models (LLMs) has dramatically enhanced the capabilities of AI agents. With the advent of LLMs such as GPT-3, GPT-4, and LLaMA, we have witnessed the achievement of human-comparable or superior performance across various tasks, including text generation, question-answering, and summarization [ 1 , 2 , 3 , 4 ] . The development of AI agents based on these LLMs has gained significant momentum, with promising applications spanning diverse domains such as customer service [ 5 ] , educational support [ 6 , 7 , 8 ] , and creative work assistance [ 9 , 10 ] . Of particular interest is whether AI agents can exhibit social behaviors similar to those of humans. Previous studies have employed various approaches to observe the social behaviors of LLM-based agents. For instance, Park et al. [ 11 ] conducted virtual daily life simulations, analyzing the behavioral patterns of 25 AI agents and their impact on a simulated society. Their study observed information sharing between agents and the formation of novel relationships. Meanwhile, Lan et al. [ 12 ] conducted research evaluating social interaction capabilities through multi-agent conversations in the board game Avalon, which requires cooperation and deception among multiple agents. Their study proposed a framework that enables AI agents to make strategic decisions based on previous gameplay experiences, reporting observations of social behaviors such as leadership and persuasion. In social interaction, verbal communication plays a central role. Previous studies on the application of LLMs have also revealed that enabling AI agents to chat with each other is an effective approach. Qian et al. [ 13 ] demonstrated that a chat chain between an instructor and an assistant is effective for completing various subtasks in the workflow of software development. Gu et al. [ 14 ] proposed a simulation framework for group chats among AI agents, reporting that multifaceted emergent behavior was observed during role-playing scenarios. Wu et al. [ 15 ] proposed a platform for LLM applications that supports interaction between LLMs, humans, and tools, where group chats among AI agents are facilitated. However, text chats are significantly different from human-to-human conversations. It has been claimed that text chat is incoherent, especially due to the lack of interaction management such as simultaneous feedback, which leads to disruption and breakdown of turn-taking and topic management [ 16 ] . Most AI chat systems employ an even simpler turn-taking model: sending text input from the user initiates the turn transition. This framework does not reflect the properties that human conversation has. For example, chat AIs cannot actively offer topics, initiate conversations, remain silent when other participants are to speak, or withhold from speaking. Turn-taking plays a crucial role especially in multi-party conversations, yet there have been relatively few studies on such conversation by AI agents. In order to handle multi-party conversations, the problem of selecting the next speaker arises. In the AutoGen platform [ 15 ] , an automatic next-speaker selection mechanism is implemented, where an LLM agent estimates the next speaker’s role based on the history of the speaker’s role and utterances. However, Bailis et al. [ 17 ] pointed out that while this approach is potentially effective, it lacks autonomy for individual agents. Instead, they proposed a dynamic turn-taking system where agents express their desire to speak by bidding. As Bailis et al. [ 17 ] argued, allowing agents to autonomously determine the speaking order could be key to AI agents playing their own social role and having a fruitful conversation. At the same time, however, the order of speaking should not be determined solely by the agents’ will. Sociologists who pioneered conversation analysis devised a concept of adjacency pairs [ 18 ] as the basic unit of utterance sequences. An adjacency pair is a two-part exchange in which the second utterance is functionally dependent on the first. Such functional binding is called conditional relevance [ 19 ] . When the current speaker addresses a question to another one, the addressee is not only obligated to take the turn, but also to speak something relevant to the question. In multi-party conversations, the first pair part of adjacency pairs often involves this “current speaker selects next” technique [ 20 ] . Therefore, the research question here is whether introducing turn-taking systematics such as adjacency pairs, discovered in the research field of conversation analysis, into the next-speaker selection mechanism will have the effect of making LLM-based multi-agent conversations more natural and efficient. Schegloff [ 21 ] argued that organization of sequences in turn-taking systematics such as adjacency pairs is the source of coherence in conversation. If so, introducing such a conversational norm into conversations by AI agents is expected to improve the coherence of conversation. To address this research question, we developed Murder Mystery Agents (MMAgents), a system where multiple AI agents play a deductive tabletop role-playing game called Murder Mystery. MMAgents consists of a self-selection mechanism for autonomous utterances and a next-speaker selection mechanism that detects the first part of adjacency pairs using LLMs to determine the next speaker. 2 Background 2.1 LLM-Based Agents With the advancement of large language models (LLMs), numerous LLM-based agents have been proposed [ 22 , 15 , 23 , 24 ] . These autonomous agents, built upon foundational models such as GPT-3 and GPT-4, are capable of executing complex tasks, engaging in assistant-like dialogue, and making decisions. The applications of LLM-based agents span diverse domains, including software development [ 13 ] , gaming environments [ 25 ] , and economic simulations [ 26 ] . Of particular interest are multi-agent systems involving multiple agents. The CAMEL framework [ 27 ] demonstrates how

Read the complete text