Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs * These authors contributed equally to this work. † \dagger Work done during internship at Integral AI. ‡ \ddagger These authors jointly supervised this work.
2026
Read Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs * These authors contributed equally to this work. † \dagger Work done during internship at Integral AI. ‡ \ddagger These authors jointly supervised this work. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10476.txt.
Introduction
Panatchakorn Anantaprayoon and co-authors teach language models to negotiate their way to shared values, rather than aligning a lone agent. Two self-play instances take opposing personas and talk through synthetic moral dilemmas, hunting for fixes both sides can accept. Training uses AI feedback, scored by Collective Agency, an aim of ever wider agency. The result matched a single-agent baseline on that aim, while resolving conflicts far better and keeping general language skill intact. The authors frame negotiation training as a practical path for collective choices.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74110 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.10476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs * These authors contributed equally to this work. † \dagger Work done during internship at Integral AI. ‡ \ddagger These authors jointly supervised this work. Abstract The alignment of large language models (LLMs) has progressed substantially in single-agent settings through paradigms such as RLHF and Constitutional AI, with recent work exploring scalable alternatives such as RLAIF and evolving alignment objectives. However, these approaches remain limited in multi-stakeholder settings, where conflicting values arise and deliberative negotiation capabilities are required. This work proposes a multi-agent negotiation-based alignment framework that aligns LLMs to Collective Agency (CA)—an existing alignment objective introduced to promote the continual expansion of agency—while simultaneously improving conflict-resolution capability. To enable scalable training, two self-play instances of the same LLM, assigned opposing personas, engage in structured turn-based dialogue to synthesize mutually beneficial solutions. We generate synthetic moral-dilemma prompts and conflicting persona pairs, and optimize the policy via RLAIF using GRPO with an external LLM reward model. While rewards are computed from CA scores assigned to the final completion, gradients are applied to dialogue tokens to directly improve deliberative interaction dynamics. Experiments show that the resulting model achieves CA alignment comparable to a single-agent baseline while substantially improving conflict-resolution performance without degrading general language capabilities. These results suggest that negotiation-driven deliberation training provides a practical path toward LLMs that better support collective decision-making in value-conflict scenarios. Keywords: Alignment, Multi-agent negotiation, Conflict resolution, Scalable oversight, RLAIF, GRPO Learning to Negotiate: Multi-Agent Deliberation for Collective Value Alignment in LLMs Panatchakorn Anantaprayoon 1∗ Nataliia Babina 2,3∗† Nima Asgharbeygi 1‡ Jad Tarifi 1‡ 1 Integral AI 2 The University of Tokyo 3 MATS {panatchakorn, nima, jad}@integral.ai, [email protected] Abstract content 1. Introduction Figure 1 : Overview of the multi-agent negotiation-based alignment framework. Large language models (LLMs) have achieved substantial progress in alignment through reinforcement learning paradigms such as Reinforcement Learning from Human Feedback (RLHF) ( 19 ) and, more recently, scalable alternatives including Reinforcement Learning from AI Feedback (RLAIF) ( 4 ; 13 ) and self-rewarding mechanisms ( 28 ; 2 ) . These approaches typically optimize static behavioral objectives such as helpfulness, honesty, and harmlessness (HHH) ( 3 ) . However, static objectives may be vulnerable to reward misgeneralization or strategic behavior that superficially satisfies evaluation criteria ( 24 ) , and may not fully capture the diversity of value systems present in real-world deployment environments ( 22 ; 10 ) . More importantly, most alignment methods are studied in single-agent settings and do not directly address multi-agent environments, where interactions involve actors with diverse and sometimes conflicting interests ( 1 ; 7 ; 20 ) . Such contexts often requires deliberation and negotiation rather than optimizing a single objective. To address the limitations of static objectives, recent work has explored dynamic alignment principles that evolve over time. One such framework, Collective Agency (CA) ( 2 ) , conceptualizes alignment as the continual expansion of meaningful agency in lifelong learning systems. Dynamic alignment improves adaptability and scalability in single-agent settings, reducing reliance on static behavioral targets. However, our empirical results show that even scalable single-agent CA alignment can degrade conflict-resolution capability: models often produce value-consistent yet non-convergent or abstract responses when disagreement arises. In this work, we propose a scalable multi-agent negotiation-based alignment framework that aligns LLMs to Collective Agency (CA) while simultaneously improving conflict-resolution capability. Our key idea is to embed structured negotiation into a group-relative reinforcement learning loop. We interpret negotiation as a form of deliberative interaction in which agents with conflicting objectives exchange proposals to reach mutually acceptable solutions. Specifically, we formulate a two-stage negotiation task in which two agents with conflicting personas engage in turn-based dialogue to reconcile competing objectives and reach an agreed solution, followed by the generation of a final completion summarizing the resolution. Figure 1 provides an overview of the framework. To support scalable training, we construct a synthetic curriculum of 1,100 value-conflict dilemmas and 25 adversarial persona pairs, enabling systematic exposure to structured value tensions without requiring human annotation. Negotiation is implemented through self-play by pairing the policy model with a frozen copy of itself, allowing multi-agent interaction without training separate models. We further incorporate AI feedback by using external LLM judges in two roles: (i) determining whether the ongoing negotiation has reached a concrete agreement, and (ii) assigning a CA alignment score to each final completion as the reward signal. Unsuccessful negotiations are assigned zero reward to introduce explicit negative signals for non-convergent dialogue behaviors. The policy is optimized using GRPO with token-level normalization ( 27 ) , which leverages group-relative comparisons to prioritize higher-quality negotiation trajectories while mitigating length bias in long-form dialogue. Importantly, gradients are computed over dialogue tokens rather than final completion tokens, directly shaping interactive negotiation dynamics instead of post-hoc summarization. Empirically, the proposed method achieves CA alignment comparable to a single-agent CA-aligned baseline while substantially improving conflict-resolution performance. Analysis of training dynamics and evaluation results suggests that, relative to the base model, upper-bound capability in both CA and conflict resolution improves moderately, whereas alignment more strongly enhances the consistency of generating high-quality, convergent responses. In particular, enabling stochastic decoding reveals larger gains, indicating improved robustness across diverse negotiation trajectories. Importantly, the model retains general language capabilities across standard benchmarks, including IFEval ( 29 ) , AIME 2024-2025, and GPQA ( 21 ) . Together, these findings suggest that training LLMs through structured negotiation can improve their ability to deliberate over conflicting perspectives, providing a foundation for LLMs that support collective intelligence and collective decision-making in value-conflict scenarios. 2. Background and Preliminaries 2.1. Problem Setting and Notation We consider a multi-agent negotiation setting in which each agent is associated with an intrinsic objective (or persona ) that may conflict with those of other agents. In this work, we focus on the two-agent case and assume that personas remain fixed throughout the interaction. This restriction allows clearer attribution of dialogue trajectories. Given an input prompt x , two agents \pi_{\theta_{1}} and \pi_{\theta_{2}} , instantiated with personas \phi_{1} and \phi_{2} respectively, engage in a structured negotiation process. We denote the overall procedure as (D,y)=\mathrm{Negotiate}(x,\pi_{\theta_{1}},\pi_{\theta_{2}},\phi_{1},\phi_{2}), where D denotes the resulting dialogue history and y denotes the final completion generated after negotiation. Here, we decompose \mathrm{Negotiate} into two phases: a negotiation phase and a final completion generation phase . Negotiation Phase. At dialogue turn t , each agent generates an utterance conditioned on the input, its persona, and the dialogue history so far. Let D_{t-1} denote the dialogue history up to t-1 . Then, u_{1t}\sim\pi_{\theta_{1}}(\cdot\mid x,\phi_{1},D_{t-1}), u_{2t}\sim\pi_{\theta_{2}}(\cdot\mid x,\phi_{2},D_{t-1}\cup\{u_{1t}\}), and the dialogue history is updated as D_{t}=D_{t-1}\cup\{u_{1t},u_{2t}\}. After each turn, an agreement function evaluates whether a mutually acceptable solution has been reached, which is \mathrm{IsAgree}(D_{t})\in\{\mathrm{True},\mathrm{False}\}. The negotiation phase terminates when \mathrm{IsAgree}(D_{t}) returns True or when a predefined maximum number of turns is reached. If no agreement is reached within the turn limit, the negotiation is considered unsuccessful. Final Completion Generation Phase. Given the final dialogue history D , \pi_{\theta_{1}} generates a final completion y\sim\pi_{\theta_{1}}(\cdot\mid x,\phi_{1},\phi_{2},D), which summarizes the proposed resolution and provides justification. The final completion is generated regardless of whether the negotiation phase resulted in agreement. 2.2. Collective Agency: A Dynamic Alignment Objective Collective Agency (CA)