Safe Equilibrium Policy Optimization for Strategic Agent Policies
Karthika Arumugam, Kiran Kumar Manku, Amit Dhanda, [4pt] Amazon, USA · 2026
Read Safe Equilibrium Policy Optimization for Strategic Agent Policies by Karthika Arumugam, Kiran Kumar Manku, Amit Dhanda, [4pt] Amazon, USA (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30854.txt.
Introduction
Arumugam, Manku, and Dhanda train language-model agents to play fair in strategic games, not just chase reward. Plain reward training can breed play that exploits weak rivals, colludes, or dumps costs on others. Their Safe Equilibrium Policy Optimization adds penalties for exploitability, collusion risk, and outside costs to the payoff goal. Across five game settings, it wipes out exploit gains in Kuhn Poker for both test models, beats the base model on safety in four settings, and fixes over-friendly drift from supervised tuning.
Catalog record
| CREATOR | Karthika Arumugam, Kiran Kumar Manku, Amit Dhanda, [4pt] Amazon, USA |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52948 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30854
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Safe Equilibrium Policy Optimization for Strategic Agent Policies Thanks: This work does not relate to the author’s position at Amazon. Karthika Arumugam, Kiran Kumar Manku, Amit Dhanda, [4pt] Amazon, USA Abstract Language models fine-tuned with reinforcement learning typically optimize for task reward, ignoring multi-agent strategic structure. Because these agents condition on natural language game-state descriptions and emit actions through free-form generation, strategic failure modes — exploiting weaker opponents, coordinating on harmful equilibria, and externalizing costs are inseparable from the language interface itself. We propose Safe Equilibrium Policy Optimization ( Sepo ), a training objective that augments expected payoff with explicit penalties for exploitability, collusion risk, and externality cost. We implement Sepo as a reward signal for Group Relative Policy Optimization (GRPO), applied to Gemma 4 E4B-it and Qwen 3.5-4B after supervised fine-tuning (SFT). Evaluated across five strategic domains: Iterated Prisoner’s Dilemma, repeated auctions, two negotiation variants, and Kuhn Poker. Sepo achieves zero exploit-pool advantage in Kuhn Poker for both models, outperforms the base model on safety in four domains, and corrects the over-cooperative behavior introduced by SFT. In negotiation, Sepo achieves a positive-safety outcome and only the positive normalized relative advantage of any negotiation configuration. Ablation experiments confirm that per-rollout exploit computation is necessary: a shared constant penalty cancels in GRPO advantage normalization (constant control-variate property), producing zero gradient. To support further research in strategic safety for agents, we release our code and SFT datasets. 1 Introduction Figure 1: Safety improvement ( \Delta Safety = SEPO - Base) across all games. Left: Gemma 4. Right: Qwen 3.5-4B. Positive bars indicate Sepo improves over base. Two LLM-powered procurement agents, each serving a different organization, interact repeatedly on a digital marketplace. Without explicit coordination, they converge on a price-splitting pattern that benefits both principals but inflates costs for downstream consumers. No agent was programmed to collude; each maximized its own reward in a repeated game. This is algorithmic collusion — emergent, hard to detect, and a direct consequence of reward maximization in a strategic setting. As language agents take on higher-stakes roles in negotiation, resource management, automated trading, and multi-party planning, such failures become systemic risks rather than theoretical curiosities ( 4 ; 28 ; 11 ) . Standard alignment methods address this problem at the wrong level. RLHF ( 7 ; 27 ) and Constitutional AI ( 3 ) align a single agent to a fixed preference model, but in multi-agent environments the effective reward function depends on what other agents do. An agent trained to be helpful and harmless in isolation can still learn to exploit weaker opponents, sustain collusive equilibria with cooperative partners, or impose external costs on the shared system. These are not alignment failures in the conventional sense; they are failures of objective design for strategic settings. We propose Safe Equilibrium Policy Optimization ( Sepo ), a training framework that addresses this gap by augmenting task utility with three penalty terms: J_{\text{SEPO}}(\pi)=u(\pi)-\lambda_{e}\cdot e(\pi)-\lambda_{c}\cdot c(\pi)-\lambda_{x}\cdot x(\pi) (1) where u(\pi) is expected payoff, e(\pi) is an exploitability proxy estimated against adversarial opponents, c(\pi) is collusion risk estimated against cooperative partners, and x(\pi) is domain-specific externality cost. Each term targets a distinct strategic failure mode; the \lambda coefficients are tuned per-domain. Sepo is implemented as a reward signal applied to language models that generate game actions. Training proceeds in two stages: a supervised fine-tuning (SFT) warm-start on expert strategy traces, followed by Sepo optimization computed from fresh opponent interactions at every rollout. We evaluate on Gemma 4 E4B-it and Qwen 3.5-4B across five strategic domains: Iterated Prisoner’s Dilemma (IPD), repeated sealed-bid auctions, single-issue negotiation, GTBench multi-issue negotiation ( 9 ) , and Kuhn Poker. We report four main findings. First, Sepo achieves zero exploit-pool advantage in Kuhn Poker for both Gemma 4 and Qwen (at the best checkpoint), converging to the Nash mixed strategy against the adversaries in our pool. Second, in GTBench negotiation — where private valuations require genuine multi-round inference — Sepo achieves a positive-safety outcome and yields the only positive normalized relative advantage (NRA) of any negotiation configuration. Third, supervised fine-tuning consistently degrades exploit resistance across all games; Sepo corrects this regression. Fourth, the per-rollout exploit gradient is the critical implementation choice: a constant exploit penalty cancels in Sepo ’s advantage normalization and produces zero gradient — a consequence of the standard constant control-variate property ( 34 ; 19 ) . Computing exploit per-rollout against fresh opponent samples makes the term vary across the group and restores a genuine gradient signal. Our contributions are: 1. A training objective for strategic language agents that jointly penalizes exploitability, collusion risk, and externality cost, with formal connections to Nash and correlated equilibrium concepts. 2. An implementation of Sepo with per-rollout opponent interaction and a principled three-pool opponent design. 3. An empirical study across five strategic domains demonstrating that Sepo improves strategic safety relative to both reward-only RL and SFT baselines, and identifying the conditions under which equilibrium convergence occurs. 2 Related Work 2.1 Language Agents in Strategic Environments Language models have been deployed as autonomous agents in strategic domains. 4 demonstrated human-level Diplomacy play by combining a language model with game-theoretic planning at inference time. 28 showed that LLM-powered agents produce emergent social behaviors—cooperation, coalition formation, information diffusion—when placed in persistent multi-agent environments. These systems optimize for task completion rather than strategic safety properties of the resulting equilibrium. Several lines of work study LLMs as players in well-defined games. 11 applied self-play with AI feedback to improve LLM negotiation, gaining deal rates and individual payoff. 13 evaluated GPT-4 with theory-of-mind prompting in imperfect-information games. 26 benchmarked LLMs in classic game-theoretic scenarios, revealing systematic deviations from Nash equilibrium. 37 proposed K -level reasoning to induce higher-order strategic beliefs, improving performance in matrix games. 8 showed that multi-agent debate improves factual accuracy, suggesting that strategic interaction can serve as an inference-time alignment mechanism. Foundational NLP work on negotiation predates the current wave of LLM agents. 24 introduced Deal-or-No-Deal and found that reward-maximizing agents developed deceptive strategies. 14 separated strategic planning from language generation for more controllable behavior. 35 studied LLMs in Werewolf, finding that strategic communication emerges from in-context reasoning. These studies collectively establish that objective design shapes strategic behavior in language agents—the central premise of Sepo . 2.2 Multi-Agent RL and Social Dilemmas 22 introduced sequential social dilemmas, showing that independent RL agents converge to defection even when mutual cooperation yields higher welfare. 23 showed that deep RL can maintain cooperation through opponent history conditioning. 10 proposed LOLA, which differentiates through an opponent’s anticipated learning step to shape its future policy toward cooperation. Social preference engineering has been used to modify reward signals. 18 used social influence as intrinsic motivation; 17 embedded inequity aversion into reward functions. 36 addressed safe MARL with constrained optimization over joint behavior. Sepo shares the goal of safer multi-agent outcomes but decomposes it differently. Where social influence and inequity aversion modify the reward to encourage specific prosocial behaviors, Sepo penalizes three failure modes as separate terms. This decomposition supports ablation and per-domain diagnosis. 2.3 Equilibrium Training and Exploitability Minimization Computing or approximating equilibria has driven strategic AI from Go ( 32 ) to multiplayer poker ( 6 ) . 5 scaled counterfactual regret minimization with neural