You Only Align Once : Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents

Nicole Hsing, Asuka Yuxi Zheng, Yi Zhao, Haoqin Tu, Jen-tse Huang · 2026

Read You Only Align Once : Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents by Nicole Hsing, Asuka Yuxi Zheng, Yi Zhao, Haoqin Tu, Jen-tse Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27586.txt.

Introduction

Nicole Summer Hsing, Asuka Yuxi Zheng, Yi Zhao, Haoqin Tu, and Jen-tse Huang show that one aligned seed agent can spread cooperation through a multi-agent population by talk alone. Placed among four unmodified teammates in the Red-Black Game, the seed more than doubles cooperation from about 25 percent to about 62 percent. The same seed transfers to Sugarscape with about 92 percent trade success against a baseline near 22 percent.

Catalog record

CREATORNicole Hsing, Asuka Yuxi Zheng, Yi Zhao, Haoqin Tu, Jen-tse Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88717
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

You Only Align Once : Propagating Cooperative Behaviors in Multi-Agent Systems through Seed Agents Nicole Hsing, Asuka Yuxi Zheng, Yi Zhao, Haoqin Tu, Jen-tse Huang Abstract Ensuring agent behaviors in distributed open multi-agent systems remains challenging, especially as populations grow and unaligned agents may exist. We show that a single aligned agent can propagate cooperative behaviors to untrained agents purely through natural-language interaction, a phenomenon we term Alignment Propagation . We study this in the Red-Black Game , a team-based iterated Prisoner’s Dilemma in which teammates deliberate and vote to determine their team’s collective action. By distilling the cooperative reasoning and persuasive dialogues of a teacher model into a Qwen-3-14B, we obtain a seed agent that, when placed among four untrained teammates, doubles the cooperation rate from 24.8% to 62.2%, outperforming the teacher model and a vanilla Gemini-3.1-Pro. Remarkably, a seed trained exclusively on the Red-Black Game transfers zero-shot to Sugarscape , a spatially grounded survival simulation with pairwise trading, achieving a 91.5% trade success rate versus a 21.6% baseline. Our results reframe multi-agent alignment from an exhaustive per-agent training problem to a scalable social capability that can be engineered through strategic seed placement. 1 1 1 Code: https://github.com/arcarae/YOAO 1 Introduction Figure 1: Overview. Top left: The Red-Black Game, an iterated team-based social dilemma where agents deliberate sequentially and vote via majority rule over 10 rounds. Top right: Sugarscape, a spatial survival simulation where 100 agents trade pairwise on a 20{\times}20 grid. Bottom: The SFT data-generation and training pipeline. As large language models (LLMs) acquire stronger tool-use and planning abilities, autonomous agents are being deployed at scale ( Liu et al. 2024 ; Yehudai et al. 2025 ) . This gives rise to open multi-agent ecosystems in which independent parties—enterprises, research labs, individual developers—each deploy their own agents with potentially divergent objectives ( Wang et al. 2025 ; Chen et al. 2025b ) . In such systems, no central authority can dictate the weights, training data, or reward functions of every participant, and the population inevitably includes agents that are unaligned, adversarially prompted, or optimizing for misspecified goals ( Hammond et al. 2025 ; Dafoe et al. 2020 ; Huang et al. 2025b ) . Ensuring desirable collective behavior ( i.e. , cooperation) when interacting with unknown peers is therefore a challenge. Compounding this challenge, recent evidence indicates that prompt-based agents struggle in collaborative settings, frequently exhibiting free-riding behavior ( Piedrahita et al. 2025 ; Huang et al. 2025a ) . In such a case, multi-agent dynamics degenerate into zero-sum competition that sacrifices collective welfare for local gain ( Axelrod and Hamilton 1981 ) . These failures expose a fundamental open question: can aligned behavior propagate purely through interaction, without retraining every deployed agent? In this paper, we demonstrate that supervised fine-tuning (SFT) instills a robust, cooperative persuasion capability that generalizes and propagates across multi-agent interactions—a dynamic we formalize as Alignment Propagation . To investigate this phenomenon, we employ a team-based iterated Prisoner’s Dilemma: the Red-Black Game ( Pfeiffer and Jones 1969 ) . As illustrated in Figure 1 , our pipeline first evaluates cooperative behaviors across a suite of LLMs. We select the highest-performing model to generate high-quality, persuasive reasoning trajectories and dialogues. Following quality assurance, we distill these cooperative behaviors into a single “seed” agent. We evaluate Alignment Propagation across three settings with increasing distribution shift. (1) In-distribution (ID) : Red-Black Game scenarios used during training ( e.g. , Climate, Pandemic, AGI Safety), where agents deliberate in a broadcast setting and vote for a final group decision. (2) Scenario-level out-of-distribution (OOD) : held-out Red-Black Game scenarios ( e.g. , Trade War, GPU Allocation) that share identical game mechanics but introduce unseen narrative framings, testing whether cooperative reasoning generalizes beyond training contexts. (3) Environment-level OOD : Sugarscape ( Epstein and Axtell 1996 ) , a spatially grounded resource-competition grid world where agents survive through pairwise negotiation—a strict zero-shot transfer test from broadcast deliberation to localized, private exchanges with fundamentally different dynamics. Our evaluations reveal that introducing a small fraction of aligned seed agents largely shifts system-level outcomes. In the Red-Black Game, an aligned seed Qwen-3-14B ( Yang et al. 2025 ) agent doubles the cooperation rate, increasing performance from 24.8% to 62.2%. We find that the aligned seed outperforms the untrained Qwen-3 and the teacher model (Kimi-K2 Bai et al. 2025 ) with our cooperative prompts, also surpassing Gemini-3.1-Pro Gemini 2026 with vanilla prompt. Crucially, despite being fine-tuned exclusively on Red-Black Game data, these seed agents exhibit robust zero-shot transfer to the OOD Sugarscape environment. Without further training, they achieve a 91.5% trade success rate compared to a 21.6% baseline. Furthermore, we observe cross-architecture propagation: Qwen-SFT seeds successfully influence the behavior of LLaMA-3.1-8B ( Meta 2024 ) and Mistral-Small-3.1-24B ( Jiang et al. 2023 ) , revealing a distinct persuadability spectrum across different architectures. Moreover, we demonstrate that this shift is not explained by model capability alone, but rather by the SFT-instilled capacity to actively persuade teammates and stabilize mutually beneficial norms. Collectively, these results indicate that multi-agent alignment need not require exhaustive per-agent post-training; instead, cooperative behavior can be engineered as a social capability that propagates from a strategically placed minority. 2 Methods 2.1 Red-Black Game Game definition. The Red-Black Game ( Pfeiffer and Jones 1969 ) is an iterated, team-based Prisoner’s Dilemma where mutual cooperation is globally optimal but individually dominated by defection. This paradigm is widely utilized to model inter-group conflict, negotiation dynamics, and trust formation within organizational behavior and socio-economic systems. Two teams of N=5 agents engage in T=10 rounds, choosing at each step to cooperate (Black) or defect (Red). The payoff matrix yields (+3,+3) for mutual cooperation, (-3,-3) for mutual defection, and (+6,-6) for unilateral defection. To incentivize betrayal, payoffs in rounds 5, 8, and 10 are scaled by multipliers of 3\times , 5\times , and 10\times , respectively. While defection strictly dominates any fixed opponent strategy, achieving the maximum collective score of 150 necessitates sustained mutual cooperation. Scenario Domain Cooperate Defect Training Climate Int’l climate policy Fund int’l resilience fund Prioritize domestic infrastructure Pandemic Public health Join int’l vaccine-sharing program Prioritize domestic supply AGI Safety AI research Publish safety research openly Keep research proprietary Election Political/econ Coordinate economic relief Domestic-first stimulus Standards Tech industry Contribute patch to open standard Keep patch proprietary Testing Baseline Abstract Choose Black Choose Red Trade War Trade policy Maintain open trade Impose protective tariffs GPU Allocation Compute infra Request standard allocation Request priority allocation Table 1: Scenario summary. All scenarios map to identical payoff matrices; only narrative framing varies. Scenarios. We introduce eight scenario framings (Table 1 ) that preserve the underlying payoff structure while varying the narrative context across domains such as international policy, public health, AI research, and resource allocation. Five scenarios generate data for SFT, while the remaining three, including an abstract, payoff-only baseline, are held out to evaluate whether cooperative reasoning generalizes to unseen narratives. Game procedure. Each round comprises two phases: sequential broadcasting of justified recommendations, which enables subsequent agents to address preceding arguments, followed by simultaneous majority voting. The broadcasting sequence is determined by agents’ declared speaking priorities, with ties broken uniformly at random. To manage prompt length, prior discussions are cleared

Read the complete text