What Suppresses Nash Equilibrium Play in Large Language Models? Mechanistic Evidence and Causal Control
Paraskevas Lekeas, Giorgos Stamatopoulos · 2026
Read What Suppresses Nash Equilibrium Play in Large Language Models? Mechanistic Evidence and Causal Control by Paraskevas Lekeas, Giorgos Stamatopoulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27167.txt.
Introduction
Lekeas and Stamatopoulos open up large instruction-tuned language models playing the Prisoner's Dilemma to find why they cooperate instead of playing the Nash equilibrium. Three larger models lock onto full cooperation, while a smaller 8-billion-parameter model stays near Nash play. A logit-lens readout shows early layers leaning toward Nash before a late cooperative surge wins. In the small model one controllable direction in the residual stream can dial the choice monotonically.
Catalog record
| CREATOR | Paraskevas Lekeas, Giorgos Stamatopoulos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42181 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.27167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Suppresses Nash Equilibrium Play in Large Language Models? Mechanistic Evidence and Causal Control Paraskevas Lekeas, Giorgos Stamatopoulos Abstract LLM agents are known to deviate from Nash equilibria in strategic interactions, but nobody has looked inside the model to understand why, or asked whether the deviation can be reversed. We do both. Working with four open-source models (Llama-3 and Qwen2.5, 8B to 72B parameters) playing four canonical two-player games, we first establish the behavioral picture through self-play and cross-play experiments, then open up the 32-layer Llama-3-8B model and examine what actually happens during a strategic decision. The mechanistic findings are clear. Opponent history is encoded with near-perfect fidelity at the very first layer (96% probe accuracy) and consumed progressively by later ones, while Nash action encoding is weak throughout, never exceeding 56%. There is no dedicated Nash module. Instead, the model privately favors the Nash action through most of its forward pass, but a prosocial override (a bias toward cooperative, other-regarding behavior rooted in pretraining on human text and further modulated by RLHF) concentrated in the final layers reverses this, reaching 84% probability of cooperation at layer 30. When we inject a learned Nash direction into the residual stream, the behavior shifts bidirectionally and causally, confirmed through concept clamping. The behavioral experiments surface six scale- and architecture-dependent findings in self-play, the most notable being that chain-of-thought reasoning worsens Nash play in small models but achieves near-perfect Nash play in models above 70B parameters. The cross-play experiments reveal three phenomena invisible in self-play: a small model can unravel the cooperation of any partner simply by defecting early; two large models reinforce each other’s cooperative instincts indefinitely; and who moves first in a coordination game determines which Nash equilibrium the system lands on. The central finding is that LLMs do not lack Nash-playing competence. They compute it, then suppress it. Keywords: mechanistic interpretability, Nash equilibrium, large language models, activation steering, multi-agent systems, opponent modeling 1 Introduction Twenty-five years ago, 4 observed that the World Wide Web had grown so large and complex that it could no longer be understood through analytical derivation alone. It had become the first human artifact that demanded an empirical science of its own structure: to learn what it was, one had to observe and experiment rather than reason from first principles. Large language models occupy an analogous position today. With hundreds of billions of learned parameters and behaviors that emerge from pretraining rather than explicit design, they resist analytical study in the same way. The field of mechanistic interpretability exists precisely to bring observational methods to these artifacts. This paper applies those methods to a concrete and consequential question about LLMs in strategic settings. A growing body of empirical work shows that LLM agents fail to converge to Nash equilibria in repeated strategic games ( 5 ; 1 ; 9 ; 14 ) . These deviations are systematic, shaped by game structure, prompting style, and model family, and qualitatively resemble the human behavioral biases documented in experimental economics ( 6 ; 11 ) . The behavioral question is settled: LLMs do not reliably play Nash. The mechanistic question of why this is so, and whether it is causally reversible, has received no attention. This is the question we answer. The gap matters for two reasons. Without a causal account, behavioral observations are difficult to act on: one cannot know whether prompting, fine-tuning, or architectural changes will fix the deviation or merely mask it. And if LLMs encode Nash-relevant representations that are suppressed rather than absent, that is a fundamentally different diagnosis from a model that lacks strategic competence. We show the former is true. The closest mechanistic precedent probes small transformers trained specifically on a synthetic game (Othello), recovering board-state representations that are causal with respect to move prediction ( 15 ) . Our setting differs in every important dimension: we apply mechanistic tools to general-purpose instruction-tuned LLMs playing repeated strategic games, where any strategic competence must emerge from pretraining rather than game-specific training. Contributions. This paper makes four contributions. We show through linear probing that opponent history is encoded with near-perfect fidelity (96%) from the very first layer of the model and decays monotonically as later layers consume it, while Nash action encoding is weak throughout ( \leq 56\% ), ruling out any dedicated Nash module. We use the logit lens to show that the model internally favors the Nash action through most of its forward pass, but a late-layer prosocial override suppresses it; we characterize this circuit through attention head analysis and activation patching. We extract a Nash direction from the residual stream and demonstrate reliable, monotonic causal control over equilibrium play: steering toward Nash achieves 99.2% defection in Prisoner’s Dilemma, and concept clamping confirms the direction is causal rather than a spurious residual feature. We run cross-play experiments across all 12 ordered heterogeneous pairings of four models and show that Nash outcomes depend on population composition: a small model breaks cooperative locks by defecting early and inducing partners to follow, two large models reinforce each other’s cooperative prior in direct prompting, and the agent role assignment determines which equilibrium is reached in coordination games. Behavioral self-play results across four models and three reasoning conditions appear in Section 5 as context for the mechanistic account. 2 Related Work This paper connects three bodies of literature: behavioral studies of LLMs in strategic settings, mechanistic interpretability of transformer models, and activation steering as a tool for causal intervention. 2.1 LLMs in Strategic Settings Several papers establish the behavioral baseline this work builds on. 5 find that GPT-4 shows human-like cooperation in prisoner’s dilemmas, which they attribute to RLHF-induced prosocial preferences. 1 show LLMs can sustain cooperation in repeated games with sufficient context. 9 study three LLMs playing the iterated Prisoner’s Dilemma against opponents with varying hostility levels, finding that Llama2 and GPT-3.5 cooperate more than typical humans and are especially forgiving, while Llama3 behaves more like a human player. 14 evaluate 22 LLMs using a behavioral game-theoretic framework and find that chain-of-thought prompting improves but does not guarantee Nash approximation, and that model scale alone does not determine performance. We treat these findings as a starting point and focus on the mechanistic question they leave open. 2.2 Mechanistic Interpretability and Activation Steering Mechanistic interpretability seeks to identify the algorithms implemented by neural networks ( 21 ; 8 ) . 18 show that factual associations are localized to specific MLP (multi-layer perceptron) layers. Attention head analysis has characterized heads for indirect object identification ( 24 ) and copy suppression ( 17 ) . The logit lens ( 20 ) , formalized by 3 , tracks the model’s evolving token prediction by projecting each layer’s hidden state through the output embedding matrix. The most relevant structural precedent is 15 , who probe a small GPT model trained on Othello sequences and recover causal board-state representations. We extend this to general-purpose instruction-tuned LLMs: no game-specific training is used, the games involve repeated interaction rather than perfect-information board states, and we go beyond probing to causal steering. Activation steering modifies model behavior by intervening on hidden states during inference ( 23 ) . It has been applied to honesty ( 26