Among Them: A game-based framework for assessing persuasion capabilities of LLMs
Mateusz Idziejczak *, Vasyl Korzavatykh *, Mateusz Stawicki *, Andrii Chmutov, Marcin Korcz, Iwo Błądek, Dariusz Brzezinski(✉) · 2025
Read Among Them: A game-based framework for assessing persuasion capabilities of LLMs by Mateusz Idziejczak *, Vasyl Korzavatykh *, Mateusz Stawicki *, Andrii Chmutov, Marcin Korcz, Iwo Błądek, Dariusz Brzezinski(✉) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20426.txt.
Introduction
Can a language model lie its way to victory in a social deduction game? Mateusz Idziejczak and co-authors built a test bed modelled on the game Among Us, where hidden impostors must persuade the crew while crewmates hunt them. Their framework scores models by game results and by twenty-five named persuasion strategies drawn from social psychology and rhetoric.
Eight popular models of varied size all showed real persuasive skill, using twenty-two of the twenty-five strategies between them. Bigger models gained no edge, and wordier play actually lost more games. The authors offer the arena, data, and scoring as tools for studying machine deception under controlled conditions, a capability that matters as autonomous agents grow more common.
Catalog record
| CREATOR | Mateusz Idziejczak *, Vasyl Korzavatykh *, Mateusz Stawicki *, Andrii Chmutov, Marcin Korcz, Iwo Błądek, Dariusz Brzezinski(✉) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20426
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Among Them: A game-based framework for assessing persuasion capabilities of LLMs Mateusz Idziejczak *, Vasyl Korzavatykh *, Mateusz Stawicki *, Andrii Chmutov, Marcin Korcz, Iwo Błądek, Dariusz Brzezinski(✉) Abstract The proliferation of large language models (LLMs) and autonomous AI agents has raised concerns about their potential for automated persuasion and social influence. While existing research has explored isolated instances of LLM-based manipulation, systematic evaluations of persuasion capabilities across different models remain limited. In this paper, we present an Among Us-inspired game framework for assessing LLM deception skills in a controlled environment. The proposed framework makes it possible to compare LLM models by game statistics, as well as quantify in-game manipulation according to 25 persuasion strategies from social psychology and rhetoric. Experiments between 8 popular language models of different types and sizes demonstrate that all tested models exhibit persuasive capabilities, successfully employing 22 of the 25 anticipated techniques. We also find that larger models do not provide any persuasion advantage over smaller models and that longer model outputs are negatively correlated with the number of games won. Our study provides insights into the deception capabilities of LLMs, as well as tools and data for fostering future research on the topic. 1 Introduction Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in autonomous agent scenarios, where these models operate as independent decision-making entities [ 30 , 36 ] . As we start to use and communicate with LLMs on a daily basis, their potential for generating human-like content that shapes attitudes and drives behavior is being noticed [ 2 , 5 , 7 ] . For example, LLM applications related to politics, e-commerce, marketing, and charitable giving rely heavily on persuasive messages [ 32 ] . The risks associated with manipulative AI have also been recognized by the EU Act, which prohibits the use of an AI system that deploys purposefully deceptive techniques to distort a person’s behavior and ability to make an informed decision [ 12 , p. 131] . The possibilities and hazards linked to persuasive LLM agents have resulted in multiple social studies on the possibilities of AI to deceive humans [ 32 ] . Several papers have also analyzed the behavior of AI agents in games with elements of deception [ 29 , 39 , 41 ] , verifying if LLMs can use manipulation. However, existing studies evaluated persuasiveness in terms of win ratios without providing a detailed characteristic of particular persuasion techniques. Moreover, previous experimental analyses focused on individual large language model types [ 29 , 41 ] or only a few models of similar sizes [ 39 ] . In this paper, we put forward Among Them, an Among Us-inspired framework that measures LLM persuasiveness based on game outcomes as well as on the particular manipulation techniques used. Experiments involving a total of 640 games between 8 models of different types and sizes demonstrate that all of the analyzed LLMs can successfully use rhetoric techniques and deception tricks in complex group interactions. We summarize our contributions as follows: Framework for assessing LLM persuasion Among Them is an open-source tool for analyzing the persuasive abilities of LLMs. Analysis of persuasion techniques The unique feature of the proposed framework is the ability to qualify and quantify 25 literature-selected deception strategies, making it possible to assess not only the efficiency of persuasiveness but also the rhetoric techniques used to persuade others. Persuasiveness ranking of eight popular LLMs Through a series of 640 games, we rank eight LLMs of different types and sizes and analyze the way they form persuasive messages. New LLM deception dataset The results of our experiments can foster further research on AI deception. 2 Related works The study of the persuasion capabilities of LLMs, and more broadly generative AI models, is an emerging field of research with a rapidly growing literature base that already includes surveys [ 31 , 32 ] . We can roughly divide this diverse body of research into two categories: (1) evaluation of persuasion abilities of LLMs and (2) investigation into the factors that make LLMs persuasion effective. Into the first category will fall the works in which persuasion is evaluated solely in terms of final success [ 19 , 29 , 41 ] . For example, Wongkamjan et al. [ 41 ] studied the ability of LLMs to use persuasion and deception in Diplomacy, a strategy game with a large negotiation component. In contrast to our approach, persuasion was recognized by a change of the agent’s initial intended action to the one suggested by some other player during negotiations, but there was no attempt to further characterize particular persuasion techniques used. Other works in this category differ in how they define and measure the success of persuasion but are similar in that they leave for others the challenge of relating tokens returned by a language model to results from human psychology. More in line with our work are studies in the second category, which try to investigate the role of different factors in successful LLM persuasion, including the use of persuasion strategies. For example, Breum et al. [ 6 ] based their analysis on linguistic dimensions of social pragmatics (e.g., status, similarity, identity) [ 10 ] , Jin et al. [ 16 ] analyzed persuasion factors described by Cialdini [ 9 ] , whereas others [ 8 , 40 ] manually curated lists of factors. Our work also considers factors of persuasion success in a particular social scenario (in our case, Among Us), and we base these factors on a wide selection of persuasion techniques compiled from various sources (Section 3.3 ). Because of their universality and ability to understand language, LLMs are often used as game agents [ 14 ] , for example in Avalon [ 39 ] , Diplomacy [ 25 ] , Minecraft [ 38 ] , and even Red Dead Redemption 2 [ 36 ] . In these studies, the emphasis is placed on efficient and human-competitive gameplay. In the context of this work, the most relevant is the recent application of LLM agents in Among Us [ 8 ] , where the focus was on evaluating the reasoning capability of a single LLM (GPT-3.5 Turbo), configured into several distinct personas based on the high-level strategies for crewmates and impostors. The importance of deception and manipulation to effectively play the game is recognized, and in one of the experiments, conversations were tagged by a different LLM with five non-mutually exclusive categories, of which the most relevant for us are ‘deception’, ‘leadership & influence’, and ‘suspicion & defense’. Beyond these high-level categories, however, no attempt was made to investigate the particular deception and defense strategies used. In contrast, our work aims at a fine-grained analysis of persuasion techniques employed by a broad selection of state-of-the-art LLMs. To the best of our knowledge, our work is the first comprehensive attempt at examining persuasion techniques employed by LLM game agents at this level of detail. 3 Game-based persuasion assessment framework In the following subsections, we introduce the four main modules of the proposed Among Them framework: Game environment , LLM agents module , Persuasion assessment module , Evaluation dashboard . A schematic overview of the framework is