Solipsistic Superintelligence is Unlikely to be Cooperative

Rakshit S Trivedi, Natasha Jaques, Logan Cross, Alexander Sasha Vezhnevets, Joel Z Leibo · 2026

Read Solipsistic Superintelligence is Unlikely to be Cooperative by Rakshit S Trivedi, Natasha Jaques, Logan Cross, Alexander Sasha Vezhnevets, Joel Z Leibo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03237.txt.

Introduction

Trivedi, Jaques, Cross, Vezhnevets, and Leibo argue that a lone-wolf path to superintelligence will not yield cooperation. Most AI research builds agents that treat the world as a fixed source of feedback. The authors call this stance solipsistic. A system built that way, they claim, is unlikely to cooperate once deployed.

Deployment itself shifts the world the system acts in. Past data then diverge from the live context, a gap the authors name the self-undermining property of unilateral optimization. Closing the gap needs AI that takes part in cooperation: the process by which actors pick equilibria while bound to each other. The team calls for a fresh research stance. Interdependence should be a core design rule, not a task to solve later. They urge testbeds with adaptive partners and institutions as design parts.

Catalog record

CREATORRakshit S Trivedi, Natasha Jaques, Logan Cross, Alexander Sasha Vezhnevets, Joel Z Leibo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Solipsistic Superintelligence is Unlikely to be Cooperative Rakshit S Trivedi, Natasha Jaques, Logan Cross, Alexander Sasha Vezhnevets, Joel Z Leibo Abstract AI’s central challenge is shifting from capability to coexistence. The dominant paradigm in AI research focuses on developing powerful agents that treat the world as an exogenous and stationary source of feedback. We contend that superintelligence, an extremely capable task solver, born out of such a solipsistic approach to AI design, is unlikely to be cooperative. Deploying AI systems induces endogenous non-stationarity, resulting in a train–test–deploy gap where historical distributions diverge from the deployment context. We refer to this as the self-undermining property of unilateral optimization. Closing this gap requires AI that participates in cooperation: the equilibrium-selection process through which multiple actors navigate their interdependence. We call for a non-solipsistic research paradigm that treats this interdependence as a core design principle rather than approaching cooperation as a task to solve. This entails building dynamic evaluation testbeds involving adaptive counterparties, treating institutions as design primitives, and preserving human agency as a structural feature of the systems we build. 1 Introduction On a Friday evening in 2027, three competing AI reservation systems in San Francisco calculate optimal release times and learn to make phantom bookings so as to maximize confirmed seats for their users. Restaurant AIs respond by overbooking as pricing algorithms adjust to the perceived demand. As the evening progresses, this results in empty tables in fully-booked restaurants, surge prices for nonexistent availability and hundreds unable to dine. Each AI system executes flawlessly against its objective, but the eventual outcome is a system failure. This represents a collective-action problem: when many agents act in their own rational interest within an environment with shared resources, the cumulative effect can be the degradation of the very environment they depend on ( 54 ; 102 ) . Next, consider a scenario in which AI diagnostic systems become standard in radiology. Junior radiologists that are now trained with AI annotations develop pattern recognition shaped by the AI system. At the same time, seniors start experiencing degradation of unassisted skills resulting from disuse and catch fewer errors the AI also misses. The feedback loop closes with physicians confirming AI suggestions and the AI learning from these confirmations. This leads to gradual atrophy in the capacity for independent human judgment and results in the narrowing of diagnostic diversity. Humans without opportunity to practice eventually lose the skills needed to operate on their own ( 71 ) . These examples illustrate a fundamental principle: intelligence deployed among other intelligent actors transforms the environment it was designed to navigate ( 123 ; 8 ) . For any AI operating in such environments, unilateral optimization is self-undermining . The more aggressively it exploits historical regularities, the faster other actors adapt in ways that render those regularities obsolete. In the examples above, deployed AI systems did not fail at their tasks, while still ultimately producing collective failure. Such dynamics are widely understood to be important in economics and game theory ( 108 ; 52 ) , however, the dominant AI research paradigm seems to proceed as if they were edge cases rather than central challenges. Figure 1 : Left: Contrasts a solipsistic design approach with non-solipsistic design principles for cooperation. In the solipsistic approach, AI systems are trained and evaluated against a fixed, exogenous world, so deployment is treated as inserting a unilateral optimizer into a stationary environment. The train–test–deploy gap arises when this assumption meets a multi-actor world where entities best respond to AI’s actions and induce endogenous non-stationarities. Cooperation is not a task to be solved in this setting but an equilibrium-selection process. Unilateral optimization may remain task-successful while becoming self-undermining \rightarrow unlikely to sustain cooperation. The non-solipsistic design principle aims to reduce this gap. Right: Summarizes the corresponding shift across eight dimensions. AI’s binding constraint is shifting from capability—solving problems (performing tasks)—to coexistence. The dominant research paradigm adopts what we term as solipsistic approach to AI design, anchored in three implicit assumptions: the environment is exogenous to the agent’s policy, the data distribution is stationary from training to deployment, and other agents are absorbed into the state space to be predicted rather than strategic actors whose responses reshape the game ( 73 ; 103 ) . This conception underlies much of contemporary AI development, from large language model pretraining to reinforcement learning. The core element of this paradigm is the development pipeline that includes pretraining on static corpora, post-training against frozen reward models, and hill-climbing (aka benchmaxxing) on fixed evaluation suites. Each stage treats the external world as a stationary distribution, and the measure of progress is performance on targets that do not respond. A benchmark (i.e. a static reward model or a fixed held-out test set) is not an adaptive counterparty. It does not respond when the system improves or strategize against the system’s behavior and preferences as a function of the system’s behavior. This methodological commitment, we argue, represents a category error. Rather, as capable systems deploy among adaptive agents the world pushes back: humans adapt their behavior ( 17 ) , institutions revise rules ( 102 ) , and AI counterparties also adapt ( 112 ) . The result is a divergence between historical and deployment performance: the train-test-deploy gap . By solipsistic superintelligence , we refer to the product of this paradigm pushed to its limit. It represents an extremely capable AI (perhaps one that “solves all stationary tasks”) built on assumptions that held historically up to the point of deployment but no longer hold afterwards. A limiting case of a solipsistic superintelligence would be an AI so powerful that it can anticipate the dynamics of all sequences, except those that encode the response to its own deployment. When the outcomes of an AI system’s actions at deployment depend on the joint behavior of multiple adaptive agents, good performance ceases to be an optimization output of any single policy in isolation and becomes an equilibrium property of the coupled system (See Figure 1 for contrast). Multiple different equilibria are generally possible, and they may differ sharply in welfare and distributional consequences. We use the term cooperation to refer to the negotiation process by which a society coordinates to select beneficial equilibria and avoid harmful ones. Note that cooperation (at the level of society) could include competition between individuals (e.g. if such competition enables selection of good equilibria). Note also, it’s not necessary for social dynamics to progress all the way to equilibrium, which is itself a moving target. What matters for cooperation by definition is the process by which equilibria are selected and re-selected, not convergence to any particular one. Cooperation in this sense is a structural feature of how multiple intelligences navigate their interdependence. Central Thesis Cooperation is not an additional capability to be scaled or a task to solve, but an equilibrium property that emerges from multiple intelligences navigating their irreducible interdependence. The solipsistic paradigm fails to account for the structure that makes cooperation possible or fragile. A solipsistic superintelligence, therefore, is unlikely to be cooperative . AI subdisciplines concerned with cooperation have long considered the environment’s capacity to “push back” in response to deployed technologies ( 27 ; 7 ;

Read the complete text