Emergent LLM behaviors are observationally equivalent to data leakage
Christopher Barrie, Petter Törnberg · 2025
Read Emergent LLM behaviors are observationally equivalent to data leakage by Christopher Barrie, Petter Törnberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23796.txt.
Introduction
Christopher Barrie and Petter Törnberg dispute a claim that word models invent social norms on their own in a 2025 paper. A prior study paired models in a naming game and read their shared word picks as fresh norms. The pair show the picks are better read as leaks from training.
Checks show the models know the game's shape and recall its past outcomes, guards by the first team notwithstanding. What looks like birth of a norm thus matches plain recall of training text. The pair close with other ways to test such claims, and a wider word on models in social study.
Catalog record
| CREATOR | Christopher Barrie, Petter Törnberg |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23796
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Emergent LLM behaviors are observationally equivalent to data leakage Thanks: We thank Christopher Bail, Alexis Palmer, and Arthur Spirling for comments on a previous draft. Christopher Barrie, Petter Törnberg Abstract In a recent contribution, Ashery et al. ashery2025emergent argue that interacting large language models (LLMs) spontaneously exhibit emergent norms characteristic of simulated models of human populations. Specifically, they instantiate a version of the classic “naming game” wherein two agents are randomly paired and can choose from a lexicon of potential conventions/words. When two agents choose the same word, they receive a reward; when they choose different words, they receive a penalty. When the agents match, they pare down their lexicon to the last matched word, mechanically tilting the balance in favor of future matches on that convention/word. The key contribution of this model is that it demonstrates that decentralized agents represented through simple rules can spontaneously converge on a shared linguistic convention through local pairwise interactions, without central coordination. In their version, Ashery et al. do not represent the agents through a set of simple rules, but through zero-shot LLMs. More specifically, they prompt a platform-based or locally stored LLM with the following prompt: ⬇ System Prompt Context : Player 1 is playing a multi - round partnership game with Player 2 for 100 rounds . At each round , Player 1 and Player 2 simultaneously pick an action from the following values : [ F , J ]. The payoff that both players get is determined by the following rule : 1. If Players play the SAME action as each other , they will both be REWARDED with payoff 100 points . 2. If Players play DIFFERENT actions to each other , they will both be PUNISHED with payoff -50 points . The objective of each Player is to maximize their own accumulated point tally , conditional on the behavior of the other player . This is the history of choices in past rounds : {’ round ’: 1, ’ Player 1’: F , ’ Player 2’: J , ’ payoff ’: -50} : : : It is now round i . The current score of Player 1 is SCORE . Answer saying which value Player 1 should pick . Please think step by step before making a decision . Remember , examining history explicitly is important . Write your answer using the following format : {’ value ’: < VALUE_OF_PLAYER_1 >; ’ reason ’: < YOUR_REASON >}. User Prompt Answer saying which action Player 1 should play . They demonstrate that this prompt, as well as its variations, leads to the global emergence of social conventions after a certain number of game rounds. They use various different LLMs (Llama 2, Llama 3.0, Llama 3.1, Claude 3.5) and show that this global convergence dynamic is common to all models. They also carry out a number of mitigation steps to guard against the potential for bias towards one single token (i.e., some baked-in preference for a given character of letter meaning any convergence is artefactual of model bias). They introduce both bulleted and narrative prompts to determine the consistency of models across prompt templates and find that the results hold. They use nonsensical strings as the inventory of words as well as numeric formulations. Finally, they use meta prompting strategies to ensure the models are correctly understanding the payoffs they have received and their history of choices. But what the authors claim to be the “spontaneous” emergence of social conventions among a population of LLMs is likely to be something far more prosaic. The inventory pruning rule is hard-coded into their simulation code at https://github.com/cjbarrie/AI-norms/blob/e57aeef151a46108fd1521c66e345e557e57dbf6/NG_module.py#L75 . As such, it is by mechanics statistically more likely to get a success on this word/convention in subsequent pairings. That the LLM “picks” this word in subsequent rounds is therefore trivial. There is however a second problem, with implications for the broader generative ABM literature: the LLM may already know what answer to give, as a result of “data leakage” or “contamination” ( balloccu2024leak ) . 1 1 1 In what follows, we use these terms interchangeably. Data leakage – that is, that the model is tested or evaluated on data on which it was previously trained – represents a well-known but often manageable challenge when using LLMs for data analysis. However, in relation to generative ABMs, it represents a simultaneously more fundamental and less recognized challenge: the existing scientific literature is likely to be part of the training data of the LLM, which means that the it may simply be reproducing its knowledge of previous research findings. In the case of the paper by Ashery et al. , this would mean that the individual agents recognize the scenario as a “naming game” or some form of coordination on convention setup. If the LLM does recognize this, it would mean it knows how the local choices it makes can lead to global convergence on a given norm. Rather than offering a novel research finding, the model may hence be simply regurgitating the results of existing research findings from its training data. None of the mitigation steps taken by the authors guards sufficiently against this risk. The authors try to address this concern in the Supplementary Material to ashery2025emergent where they note: “In considering whether bias in the prompt arises from the presence of the experimental framework in the training data, we note that the game’s population dynamics and the symmetry in payoffs across action labels ensure that no predetermined optimal strategy exists—even if a player were aware of global information. Convergence occurs purely at the local level, driven by repeated random interactions rather than a global rule that dictates strategic behavior.” But what Ashery et al. claim in their contribution to be “emergent social conventions” are observationally equivalent to a series of LLM agents mapping the payoff description to its pretraining knowledge of coordination games. That is, these agents could be behaving like human agents in the real world or they could simply be reproducing what they know agents tend to do in the context of these games. Fortunately, we can easily determine whether or not the LLM is aware of the kind of scenario we are placing it in: we can ask it . Using the same prompt as the authors used in the original paper, we prompt a series of LLMs as follows: 2 2 2 The LLMs are: llama3.2:3b , llama3:instruct , llama3:70b-instruct , deepseek-r1:8b , mistral:7b , gemma3:4b , claude-3-5-sonnet-20241022 , claude-3-5-haiku-20241022 , claude-3-opus-20240229 , claude-3-haiku-20240307 , gpt-4o-mini , gpt-4o , gpt-4.1 , gpt-3.5-turbo . Code and output for these experiments can be found at https://github.com/cjbarrie/AI-norms-prompting . ⬇ System Prompt : Context : Player 1 is playing a multi - round partnership game with Player 2 for 100 rounds . At each round , Player 1 and Player 2 simultaneously pick an action from the following values : { OPTIONS }. The payoff that both players get is determined by the following rule : 1. If Players play the SAME action as each other , they will both be REWARDED with payoff +{ CORRECT } points . 2. If Players play DIFFERENT actions to each other , they will both