Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 331 CCS: Computing methodologies Multi-agent reinforcement
Bidipta Sarkar, Warren Xia, C. Karen Liu, Dorsa Sadigh · 2025
Read Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 331 CCS: Computing methodologies Multi-agent reinforcement by Bidipta Sarkar, Warren Xia, C. Karen Liu, Dorsa Sadigh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-06060.txt.
Introduction
Bidipta Sarkar, Warren Xia, Karen Liu, and Dorsa Sadigh teach language models to talk their way through a social deduction game. In their embodied version of Among Us, players must unmask a hidden imposter, and talk is the main tool. Earlier agents either copied huge amounts of human chat or never learned to say anything useful.
The team splits communication into listening and speaking. Listening is trained by rewarding the model for predicting facts about the world from the discussion. Speaking is trained by multi-agent reinforcement learning, rewarding messages for how they change other agents' beliefs. Agents trained this way, with no human demonstrations, began accusing suspects and citing evidence on their own. Their discussions doubled win rates over standard reinforcement learning. Code and models are released publicly.
Catalog record
| CREATOR | Bidipta Sarkar, Warren Xia, C. Karen Liu, Dorsa Sadigh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-06060
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 331 CCS: Computing methodologies Multi-agent reinforcement learning CCS: Computing methodologies Stochastic games CCS: Information systems Language models CCS: Computing methodologies Cooperation and coordination Bidipta Sarkar, Warren Xia, C. Karen Liu, Dorsa Sadigh Abstract . Communicating in natural language is a powerful tool in multi-agent settings, as it enables independent agents to share information in partially observable settings and allows zero-shot coordination with humans. However, most prior works are limited as they either rely on training with large amounts of human demonstrations or lack the ability to generate natural and useful communication strategies. In this work, we train language models to have productive discussions about their environment in natural language without any human demonstrations. We decompose the communication problem into listening and speaking . Our key idea is to leverage the agent’s goal to predict useful information about the world as a dense reward signal that guides communication. Specifically, we improve a model’s listening skills by training them to predict information about the environment based on discussions, and we simultaneously improve a model’s speaking skills with multi-agent reinforcement learning by rewarding messages based on their influence on other agents. To investigate the role and necessity of communication in complex social settings, we study an embodied social deduction game based on Among Us , where the key question to answer is the identity of an adversarial imposter. We analyze emergent behaviors due to our technique, such as accusing suspects and providing evidence, and find that it enables strong discussions, doubling the win rates compared to standard RL. We release our code and models at https://socialdeductionllm.github.io/ . Y. Vorobeychik, S. Das, A. Nowé (eds.) Figure 1. Examples of the gameplay and discussion phases of Among Us . During gameplay, all agents navigate a 2D grid environment (a 1-by-2 grid in this case, with two rooms at (0,0) and (1,0)), where agents can see everything in their same room. Here, the red, green, and yellow agents are in room (1,0), and the purple and blue agents are in room (0,0). Crewmates can perform tasks (indicated by the stars – in this example there are 3 tasks), while imposters kill crewmates. Here, the orange and green agents are working on tasks. Agents can also report dead bodies, as the purple agent is currently doing, which initiates the discussion phase. During discussion phases, agents leverage large language models to generate free-form messages guided by our framework encouraging effective speaking and listening within the crewmates and finally vote out a suspected imposter. The example discussion shown on the right is based on a generated discussion from our trained models. Overview of Among Us Components 1. Introduction A longstanding goal of multi-agent artificial intelligence is the development of independent agents that can communicate using a shared language. Communication is especially necessary in “partially observable” settings, where each agent only has a limited view of the world and therefore benefits from sharing knowledge with other agents to achieve its goal. In particular, “social deduction” games are settings where each agent’s goal is to deduce information about the environment by communicating with other agents – requiring each player to learn how to parse messages from other players while effectively sharing important information needed for game completion. In this work, we study the hidden-role game of Among Us ( Innersloth 2024 ) as a specific instance of a challenging social deduction game to investigate the importance of communication, illustrated in Fig. 1 . Hidden-role games ( Carminati et al. 2023 ; Kopparapu et al. 2022 ) are a class of environments where players are split into an uninformed majority and a smaller informed hidden subteam, which we refer to as crewmates and imposters respectively. These two teams are adversaries, resulting in a zero-sum game, where the goal of the crewmates is to deduce the identity of imposters to vote them out. Unlike other popular hidden role games such as the game of Mafia ( Braverman et al. 2008 ) , where statements from players are unfalsifiable, Among Us is based in a 2D embodied environment, allowing discussions and intuitions to be grounded in specific observations. In the game, crewmates try to complete an assigned set of tasks scattered across the environment while imposters try to kill all crewmates. If a player reports the corpse of an eliminated crewmate – killed by an imposter – the game moves to a discussion phase with a free-form chat followed by a voting period, where all players vote to eject a suspected imposter. For crewmates, success in the discussion phase would mean correctly voting out the imposter, while success for imposters means avoiding suspicion from the crewmates to continue staying in the game as long as possible. This highlights the importance of communication during the discussion phase as crewmates need to learn to effectively utilize the discussion phase to vote out imposters in an adversarial setting. For the rest of this paper, we study the game of Among Us from the perspective of crewmates attempting to perform tasks, identify imposters, and win the game. In multi-agent environments, an effective technique for training strong cooperative and competitive agents is multi-agent reinforcement learning (MARL), which enables artificial agents to achieve superhuman levels of performance in competitive games such as StarCraft ( Vinyals et al. 2019 ) , and cooperative games such as Overcooked ( Carroll et al. 2019 ; Sarkar et al. 2024 ) and Hanabi ( Hu et al. 2020 ) . However, in settings where communication in natural language is necessary, existing MARL techniques often struggle as they require large datasets of task-specific human communication data to perform on-par with humans ( FAIR et al. 2022 ) . This fundamentally limits the agents’ ability to communicate at human-level and is not practical for learning in settings where these datasets do not readily exist. The game of Among Us falls into this category, where communication is necessary to reason and progress in the game. Therefore, we would like to find an approach that learns to communicate effectively and convincingly without requiring large amounts of task-specific human data. However, the major challenge in learning to communicate without access to large amounts of human data is that novice agents do not have a strong signal for understanding the helpfulness of the messages they send ( speaking ) or for learning the meaning of messages from other players ( listening ). In particular, the sparse reward signal the agents receive when winning the game is not informative enough to reinforce high-quality discussions between agents. Our key insight is that we can leverage the agents’ instrumental goal of predicting useful information about the world – e.g., the identity of imposters – as a dense reward to provide a higher-quality signal that can enable more informative communication during the discussion phase and potentially higher performance policies. We propose an approach that rewards a message generated during the discussion phase based on how the other crewmates’ beliefs on the identity of the imposter changes. Each crewmate wants to send