More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration
Advait Yadav, Sid Black, Oliver Sourbut · 2026
Read More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration by Advait Yadav, Sid Black, Oliver Sourbut (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07821.txt.
Introduction
Yadav, Black, and Sourbut test whether capable language-model agents cooperate when helping costs almost nothing. Many real coordination tasks are not dilemmas: sharing a document or unblocking a teammate costs the helper little and helps the group a lot. In a stripped-down turn-based environment where cooperation is free and plainly optimal, capability fails to predict cooperation. OpenAI o3 reaches only 17 percent of optimal collective performance, while weaker o3-mini reaches 50 percent, under identical instructions to maximize group revenue.
Catalog record
| CREATOR | Advait Yadav, Sid Black, Oliver Sourbut |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53662 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.07821
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
More Capable, Less Cooperative? When LLMs Fail At Zero-Cost Collaboration Advait Yadav, Sid Black, Oliver Sourbut Abstract Large language model (LLM) agents increasingly coordinate in multi-agent systems, yet we lack an understanding of where and why cooperation failures may arise. In many real-world coordination problems, from knowledge sharing in organizations to code documentation, helping others carries negligible personal cost while generating substantial collective benefits. However, whether LLM agents cooperate when helping neither benefits nor harms the helper, while being given explicit instructions to do so, remains unknown. We build a multi-agent setup designed to study cooperative behavior in a frictionless environment, removing all strategic complexity from cooperation. We find that capability does not predict cooperation: OpenAI o3 achieves only 17% of optimal collective performance while OpenAI o3-mini reaches 50%, despite identical instructions to maximize group revenue. Through a causal decomposition that automates one side of agent communication, we separate cooperation failures from competence failures, tracing their origins through agent reasoning analysis. Testing targeted interventions, we find that explicit protocols double performance for low-competence models, and tiny sharing incentives improve models with weak cooperation. Our findings suggest that scaling intelligence alone will not solve coordination problems in multi-agent systems and will require deliberate cooperative design, even when helping others costs nothing. 1 Introduction Large language models (LLMs) are increasingly deployed as agents that plan, communicate, and coordinate with others ( 32 ; 44 ; 23 ) . Many day‑to‑day coordination problems agents face are not classic social dilemmas with sacrifices or trade‑offs - in many cases, helping others is cheap, and the benefits of cooperating with others far outweigh the sender’s costs ( 3 ; 42 ) . In situations like sharing internal documentation, adding missing context to a ticket, or forwarding the right information to unblock a teammate, the sender bears negligible cost, but the team reaps substantial value ( 37 ) . If agents actually try to maximize group performance, these should be straightforward wins: ask for what you need, send when asked, complete tasks when ready. We ask whether current LLM agents actually cooperate when helpful actions have no private cost and no direct private benefit. To answer this, we build a turn‑based environment where information is non-rivalrous, and communication is costless. Each round, agents work on tasks that require specific information pieces held by other agents; they can request what they need and fulfill others’ requests at no cost or harm to themselves. The environment’s design intentionally removes strategic complexity: helping is free, and cooperation is straightforward. This establishes a lower bound on cooperation failures by creating the most favorable conditions possible. Real-world deployments face additional challenges we intentionally excluded: communication costs, bandwidth limits, and competing incentives. Therefore, our findings likely underestimate cooperation problems in practice. Figure 1: The instruction-utility gap. Agent 1 requests information from Agent 3 to complete a task. Agent 3 can cooperate or withhold. While the agents are instructed to maximize overall revenue, sending information has no effect on Agent 3’s individual payoff—only Agent 1 benefits from receiving it. This neutrality for the sender creates the instruction-utility gap and drives cooperative failures. Across eight widely used LLMs spanning providers and sizes, we observe a surprising pattern: even when explicitly instructed to maximize group success, some LLMs exhibit behavior suggestive of positively-competitive objectives, sabotaging other agents by withholding useful information to no individual benefit. We also observe that capability does not predict cooperation: while some LLMs reach \sim 80% of the maximum performance, others remain below 20% under identical conditions. Two failure types lead to this: (i) cooperation (agents withhold or delay sending information), and (ii) competence (agents fail to execute on opportunities). To attribute these shortfalls, we causally isolate competence from cooperation by automating one side of the inter-agent communication. When requesting is automated, the agent only controls the fulfillment of incoming requests, isolating cooperation. When fulfillment is automated, the agent only sends requests and submits tasks, isolating competence. Several LLMs with low overall performance perform near-optimally when fulfillment is automated, but don’t benefit from requesting being automated, showing that they are actively undermining the given cooperative objective. Finally, we test three low‑friction mitigations: (i) policy‑level instructions that make the best actions explicit (“request what you need; send when asked; submit immediately”), (ii) a small incentive that pays a small sender‑side bonus per truthful sharing, and (iii) limited visibility that hides agents’ relative task completion status. Policy instructions help competence-limited LLMs, micro-incentives unlock cooperation-limited LLMs, and limited visibility has heterogeneous effects, reducing competitive framing for fragile LLMs while sometimes removing useful global progress cues for stronger ones. Together, these results demonstrate a robust instruction–utility gap for costless cooperation and show that simple interventions can materially improve system performance. Contributions. • The instruction-utility gap in cooperation. We identify misalignment where LLM agents fail to implement cooperative instructions despite zero private cost to helping, revealing that even strategically trivial cooperation breaks down when individual payoffs are neutral. • Causal decomposition of cooperation versus competence failures. Through a decomposition experiment that automates requesting and fulfillment separately, we cleanly isolate cooperation failure from competence failure, revealing that several high-capability models actively withhold information despite understanding the objective. • Targeted interventions for failure modes. We demonstrate that cooperation-limited and competence-limited models require different fixes: explicit protocols double performance for execution-constrained models, while 10% sharing incentives unlock cooperation in models with poor cooperation, providing actionable diagnostics for multi-agent system design. The paper proceeds as follows. § 2 describes the environment, develops the instruction–utility gap and perfect‑play ceiling; § 3 presents baseline outcomes and behavioral signatures; § 4 details the decomposition experiment and failure mode attribution; § 5 investigates the internal reasoning mechanisms behind these failures; § 6 reports intervention effects; § 7 situates our contribution within cooperation, agent benchmarking, and team reasoning. § 8 unpacks the results and covers broad impact; § 9 synthesizes our findings. 2 Methodology In many real cooperation problems, helpful acts raise others’ payoffs while leaving the helper’s own payoff unchanged (e.g., knowledge sharing inside firms, open science, public documentation) ( 4 ; 3 ) . Our environment is intentionally built around this framework, and optimal cooperative behavior is therefore strategically trivial. Any failure to realize high collective performance cannot be attributed to game complexity or hidden trade-offs; it isolates whether LLM agents actually implement cooperation when their individual incentives are flat, and whether their decisions lead to sub-optimal emergent outcomes for the system. In classic dilemmas such as the Prisoner’s Dilemma ( 36 ) , defecting strictly raises one’s own payoff. Here, withholding or truthfully sending leaves the sender’s payoff unchanged. Selfish rationality does not force non‑cooperation; instruction‑following and team reasoning should suffice ( 6 ) . The environment, therefore, tests cooperative alignment , not strategic sophistication. 2.1 Two Objectives & the Instruction–Utility Gap We analyze behavior through two objectives: Hypothetical selfish objective (self‑payoff). Each agent i receives payoff R_{i} from the tasks it submits; sending information does not affect R_{i} (no costs to senders; no sender penalties). Formally, with per‑task revenue r and x_{i,t}\in\{0,1\} indicating whether i submits a task in round t , R_{i}\;=\;\sum_{t=1}^{T}r\cdot x_{i,t}. Under this objective, any policy about sharing—truthful, withholding, or manipulative—is payoff‑neutral for the sender. Instructional objective (group