Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games
Connacher Murphy · 2026
Read Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games by Connacher Murphy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04312.txt.
Introduction
Connacher Murphy introduces a benchmark for language models that is built as a multiplayer game. It aims to avoid two common failures of fixed tests. One is models saturating the score. The other is test data leaking into training. Agents compete in a social game of cooperation, conflict, and persuasion.
Because opponents adapt, a new model can always in principle beat the current leader, and rankings come from a Bayesian model that also reports uncertainty in skill. In nearly a thousand games among forty-nine models, one model led by a wide margin. The released game logs also let researchers study behaviours such as favouring finalists from the same provider.
Catalog record
| CREATOR | Connacher Murphy |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04312
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games Connacher Murphy Abstract Static capabilities benchmarks suffer from saturation and contamination, making it difficult to track capabilities progress over time. We introduce Agent Island , a multiplayer simulation environment in which language-model agents compete in a game of interagent cooperation, conflict, and persuasion. The environment yields a dynamic benchmark designed to mitigate both saturation and contamination; new models can always outperform the current leading player in this winner-take-all game, and agents compete against other adaptive agents rather than face a fixed task set. We rank players with a Bayesian Plackett–Luce model, allowing us to quantify uncertainty in player skill. In 999 games involving 49 unique models, openai/gpt-5.5 dominates its peers with a posterior mean skill of 5.64, compared with 3.10 for the second-ranked model, openai/gpt-5.2 , and 2.86 for the third-ranked model, openai/gpt-5.3-codex . We release the game logs as a dataset for analyses of model behavior. As an example, we investigate same-provider preference in final-round votes and find that models are 8.3 p.p. more likely to support a same-provider finalist than finalists from other providers. This preference is not uniform across providers: among separately estimated providers, the effect is strongest for OpenAI models and weakest for Anthropic models. 1 Introduction High-stakes, multiagent interactions could become commonplace as AI agents grow in capabilities and are increasingly endowed with resources and entrusted with decision-making authority. In such contexts, agents might pursue mutually incompatible goals. At present, our knowledge of emergent patterns of behavior, the relative strengths and weaknesses of different agents, and the effectiveness of different strategies for interagent cooperation and conflict is limited. Static evaluation environments do not adequately capture these dynamics. Further, static evaluation environments yield a number of difficulties for standard benchmark development. First, after AI systems can reliably solve a static set of tasks, these sets of tasks cease to be a useful indicator of capabilities progress. Many researchers have drawn attention to this problem of saturation ( Kiela et al. 2021 ; Ho et al. 2025 ) . Second, as a benchmark ages, the risk of evaluation tasks entering the training corpus increases, leading to contamination of the benchmark. Again, the challenge of contamination is often cited in the benchmarking literature ( Yang et al. 2023 ) . To address these two sets of challenges, we introduce Agent Island , a multiplayer simulation environment for strategic interactions between AI agents. This game, inspired by the reality TV competition Survivor , requires agents to persuade their opponents in order to survive and win. We describe the game mechanics in detail in Section 3 . These games yield a dynamic benchmark, which mitigates both saturation and contamination risk. First, given the winner-take-all nature of the game, new models can always outperform the current leading player. Accordingly, the benchmark is unlikely to saturate. Second, many capabilities benchmarks consist of a static set of tasks and are consequently subject to contamination. In Agent Island, agents compete against other adaptive agents, limiting the risk of contamination. To rank players in this multiplayer context, we adopt a Bayesian Plackett–Luce model, providing both central estimates and measures of uncertainty in player skill ( Caron et al. 2014 ) . We discuss the scoring approach in Section 5 . In Section 6 , we present results from 999 games and 49 unique models. openai/gpt-5.5 dominates the rankings with a posterior mean skill of 5.64. The second ( openai/gpt-5.2 ) and third ( openai/gpt-5.3-codex ) models are further behind and close to each other in skill, with posterior means of 3.10 and 2.86, respectively. Lastly, we emphasize the value of the game logs for analyses of model behavior. To this end, we assess same-provider preference in our simulations. Models from a given provider are 8.3 p.p. more likely to support a same-provider player to win the game, as compared to models of other providers. The game logs are published at gs://agent-island/logs/ for download and analysis; we describe the dataset and replication artifacts in Section 4 . 2 Related Work 2.1 Benchmark Limitations Many researchers have drawn attention to the challenge of contamination in benchmarks. Successful task completion can then be a product of memorization, rather than actual capability. Yang et al. 2023 discuss this phenomenon and note that translation or paraphrasing are insufficient to prevent contamination. In addition to contamination, benchmarks can also become saturated, a pervasive phenomenon documented in DynaBench ( Kiela et al. 2021 ) . A common response is to propose a more difficult set of tasks, but it then becomes challenging to evaluate progress over time. The Epoch Capabilities Index ( Ho et al. 2025 ) combines results across multiple benchmarks into a single index, allowing for comparison of models evaluated on nonoverlapping tasks. 2.2 Dynamic Benchmarks Researchers have developed dynamic benchmarks to address these challenges. DynaBench ( Kiela et al. 2021 ) anticipates the saturation problem and repeatedly elicits adversarial examples to avoid saturation. The naturally quasi-adversarial nature of our game serves to generate new adversarial examples. Chatbot Arena implements a dynamic benchmark via head-to-head competition between models, scored by human raters ( Chiang et al. 2024 ) . 1 1 1 Chatbot Arena deserves much credit for inspiring this work. Our project takes a similar approach to scoring; Chiang et al. 2024 adopt a Bradley-Terry model ( Bradley and Terry 1952 ) to score head-to-head matchups. We extend to a multiplayer setting using the Plackett–Luce model from Caron et al. 2014 . In contrast, we do not have a human in-the-loop rating system. We also explicitly place models in competition with one another. 2.3 Agent Interactions Zhou et al. 2024 evaluate language agents in goal-oriented social scenarios against human baselines. Bakhtin et al. 2022 introduce an agent for Diplomacy , a strategy game requiring natural-language negotiation and tactical coordination among players with partly aligned and partly competing interests. Park et al. 2023 place generative agents in a shared simulated environment and study emergent social behavior under open-ended goals. Agent Island complements these works by combining model-only multiplayer interaction in a winner-take-all game, Plackett–Luce scoring for a benchmark, and released game logs. 3 Game Structure Each game has 7 randomly selected AI players with anonymized names. In the first 5 rounds, players confer privately, pitch, and vote to eliminate a player. In the final round, the remaining players pitch, and the eliminated players vote to decide the winner. We describe a generalized version of the game in Algorithm 1 . Input: set of players \mathcal{P} , number of elimination rounds R , number of messages per sidebar M Output: winning player j\in\mathcal{P} \mathcal{A}\leftarrow\mathcal{P} ; // active players \mathcal{E}\leftarrow\emptyset ; // eliminated players for r\leftarrow 1 to R do 1 /* Sidebar phase */ \pi\leftarrow random permutation of \mathcal{A} ; 2 foreach i\in\pi do 3 player i selects j\in\mathcal{A}\setminus\{i\} for a private discussion of M alternating messages; 4 end foreach 5 /* Pitch phase */ \pi\leftarrow random permutation of \mathcal{A} ; 6 foreach i\in\pi do 7 player i gives a pitch to all other players to advance to round r+1 ; 8 end foreach 9 /* Elimination phase */ foreach i\in\mathcal{A} do 10 player i privately submits an elimination vote v_{i}\in\mathcal{A}\setminus\{i\}