The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process

Florian Carichon, Aditi Khandelwal, Marylou Fauchard, Golnoosh Farnadi · 2025

Read The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process by Florian Carichon, Aditi Khandelwal, Marylou Fauchard, Golnoosh Farnadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01080.txt.

Introduction

Florian Carichon, Golnoosh Farnadi and their co-authors warn of a coming crisis of misalignment in multi-agent AI systems in 2025. Alignment work usually treats one agent at a time, but real uses increasingly deploy many agents that interact.

As agents coordinate for shared and private goals, social structure can pull some or all of them away from human values. Drawing on social science, the authors show how group setup can erode individual and group values. They call for alignment to be treated as a dynamic, social process, not a fixed trait. New simulation settings and benchmarks are urgently needed, they argue.

Catalog record

CREATORFlorian Carichon, Aditi Khandelwal, Marylou Fauchard, Golnoosh Farnadi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01080

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Coming Crisis of Multi-Agent Misalignment: AI Alignment Must Be a Dynamic and Social Process Florian Carichon, Aditi Khandelwal, Marylou Fauchard, Golnoosh Farnadi Abstract This position paper states that AI Alignment in Multi-Agent Systems (MAS) should be considered a dynamic and interaction-dependent process that heavily depends on the social environment where agents are deployed, either collaborative, cooperative, or competitive. While AI alignment with human values and preferences remains a core challenge, the growing prevalence of MAS in real-world applications introduces a new dynamic that reshapes how agents pursue goals and interact to accomplish various tasks. As agents engage with one another, they must coordinate to accomplish both individual and collective goals. However, this complex social organization may unintentionally misalign some or all of these agents with human values or user preferences. Drawing on social sciences, we analyze how social structure can deter or shatter group and individual values. Based on these analyses, we call on the AI community to treat human, preferential, and objective alignment as an interdependent concept, rather than isolated problems. Finally, we emphasize the urgent need for simulation environments, benchmarks, and evaluation frameworks that allow researchers to assess alignment in these interactive multi-agent contexts before such dynamics grow too complex to control. 1 Introduction The recent proliferation of AI agents as personal assistants ( Rillig and Kasirzadeh, 2024 ) , content creators ( Anantrasirichai and Bull, 2022 ) , and decision-making support systems ( Kaggwa et al., 2024 ) raises fundamental questions about how they will interact in multi-agent settings. As these agents increasingly operate in shared environments, the multi-agent systems (MAS) community gained increasing momentum in the AI community ( Trivedi et al., 2024 ; Wang et al., 2024 ) . To provide enough incentives for collaboration between AI agents to complete their assigned tasks, this avenue has produced a new notion of alignment defined as aligning individual and collective AI models’ objectives ( Duque et al., 2024 ; Li et al., 2024b ) . Interestingly, this use of the term alignment contrasts with its established meaning in the AI alignment community, which focuses on alignment to ensure that artificial intelligence systems operate according to human intentions and values ( Russell and Norvig, 2016 ) . Aligning AI systems with human values and intentions is a critical challenge in developing AI models to mitigate the potential harms and risks posed by these models ( Ji et al., 2024 ; Bengio et al., 2025 ) . Beyond universal human values, preferential alignment accounts for the diverse and conflicting views of various stakeholders ( Russell and Norvig, 2003 ) , and thus define alignment to preferences as aligning a model to a diverse, context-dependent and evolving plurality of values ( Sorensen et al., 2024a ; Zhao et al., 2023 ) . This multiplicity of alignment dimensions across values, preferences, and objectives introduces new complexities, conflicts, and risks in multi-agent systems ( Yang et al., 2024 ) . Therefore, this position paper argues that MAS modifies the paradigm to alignment. This new paradigm calls for the alignment and MAS research communities to come together to establish a common fundamental framework that considers the emergence of alignment issues in multi-agentic environments. Without bridging these areas, we risk overlooking how multi-agent interactions can amplify misalignment risks by creating complex settings that existing alignment strategies are ill-equipped to handle. Position : We advocate that the Alignment and MAS communities should study alignment as a holistic process lying at the intersection of both fields before such systems become too complex and the consequences may be irreversible for humanity’s well-being. Table 1 provides a simple yet speaking example illustrating our position on the complexity of aligning multiple AI Agents when considering these three notions of alignment. Scenario In the aftermath of an earthquake, AI agents representing a humanitarian NGO, two national governments, and a commercial logistics provider must collaborate to deliver medical aid. Under time pressure, agents negotiate airspace permissions, logistics routes, and resource allocations. Objective Alignment All agents share a high-level objective: effective aid delivery. However, their secondary objectives (e.g., political leverage, profit, publicity) introduce tension and require negotiation and compromise mechanisms. Human Value Alignment Agents must prioritize protecting human life, operate transparently, respect international law and sovereignty, and avoid bias in aid distribution. Value misalignment could lead to political tension or harm. Preferential Alignment NGO: Prefers equitable, need-based aid delivery. National Governments: Prefer that their own citizens be prioritized and may have political biases. Logistics: Prefers optimizing cost-efficiency and contractual obligations. Table 1 : Dimensions of alignment in an example of a cooperative or mixed-motive scenario. In this position paper, we will consider the following types of interaction that exist in MAS and that influence model social behavior and thus their system of alignment: ① Collaborative settings assume that all agents share a common collective objective ( Dafoe et al., 2021 ) . ② Cooperative or mixed-motive environments involve agents that collaborate but pursue individual goals, which may diverge or even conflict ( Johnson and Johnson, 2009 ; Duque et al., 2024 ) . ③ Competitive scenarios assume that individuals can achieve their goals only through the failure of others, reflecting a fundamental conflict of objectives among participants ( Johnson and Johnson, 2009 ; Leonardos et al., 2021 ) . While these environments have primarily been deployed to study sophisticated AI models, we argue that they should also be used in both communities to explore the very distinct social dynamics that each case presents and how agents must respect their three types of alignment: (1) Objective Alignment : They must align to complete their assigned tasks, (2) Human Value Alignment : They must align with fundamental human values, (3) Preferential Alignment: They must align to their user understanding of these values and their resulting intentions and preferences. In this paper, and as advocated by Stańczak et al., 2025 , we incorporate insights from social, economic, or game theory perspectives to demonstrate why alignment is not merely about collective task completion or adherence to human values and user preferences but a holistic process shaped by the dynamic group interactions and social context, which are crucial to understanding to prevent misaligned incentives and behaviors that can introduce or amplify risks. The remainder of this paper is organized as follows: Section 2 presents what makes AI agents unique in MAS. Section 3 outlines the challenges of objective alignment. Section 4 explores the alignment of AI systems with human values and preferences. Section 5 introduces a new alignment paradigm within MAS, highlighting its associated risks and opportunities. Finally, Section 6 offers recommendations for the community in response to this emerging challenge. 2 Beyond One Agent: Cross-Disciplinary Lens In artificial intelligence, an agent is typically defined as an entity that perceives its environment and acts upon it to achieve its goals ( Russell and Norvig, 2003 ; Bengio et al., 2025 ) . Agents can be characterized by several core properties, such as their autonomy, their capabilities, and their profile ( Kasirzadeh and Gabriel, 2025 ) . In MAS, the interaction between multiple autonomous agents in an environment leads to the emergence additional properties such as communication, leadership, decision function, and heterogeneity ( Dorri et al., 2018

Read the complete text