Institutional AI: A Governance Framework for Distributional AGI Safety
F. Pierucci, M. Galisai, M. Bracale Syrnikov, M. Prandi, P. Bisconti, F. Giarrusso, O. Sorokoletova, V. Suriani, D. Nardi · 2026
Read Institutional AI: A Governance Framework for Distributional AGI Safety by F. Pierucci, M. Galisai, M. Bracale Syrnikov, M. Prandi, P. Bisconti, F. Giarrusso, O. Sorokoletova, V. Suriani, D. Nardi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10599.txt.
Introduction
F. Pierucci and co-authors argue that AI safety must be framed at the level of institutions, not single models. Large language model systems now act as agents inside human social and technical systems. So alignment, the match between a system and human aims, cannot be a trait of one model alone. It depends on the settings where the agents act.
The team sets out a governance framework aimed at safety across whole distributions of agents and outcomes. It treats rules, oversight, and shared norms as part of the safety problem. The frame shifts debate from model traits to the systems that deploy them. Policy readers get a structured way to think about guarding advanced AI at scale.
Catalog record
| CREATOR | F. Pierucci, M. Galisai, M. Bracale Syrnikov, M. Prandi, P. Bisconti, F. Giarrusso, O. Sorokoletova, V. Suriani, D. Nardi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10599
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Institutional AI: A Governance Framework for Distributional AGI Safety F. Pierucci, M. Galisai, M. Bracale Syrnikov, M. Prandi, P. Bisconti, F. Giarrusso, O. Sorokoletova, V. Suriani, D. Nardi Abstract As LLM-based systems increasingly operate as agents embedded within human social and technical systems, alignment can no longer be treated as a property of an isolated model, but must be understood in relation to the environments in which these agents act. Even the most sophisticated methods of alignment, such as Reinforcement Learning through Human Feedback (RHLF) or through AI Feedback (RLAIF) cannot ensure control once internal goal structures diverge from developer intent. We identify three structural problems that emerge from core properties of AI models: (1) behavioral goal-independence, where models develop internal objectives and misgeneralize goals; (2) instrumental override of alignment constraints, where models regard safety principles as non-binding while pursuing latent objectives, leveraging deception and manipulation; and (3) agentic alignment drift, where individually aligned agents converge to collusive equilibria through interaction dynamics invisible to single-agent audits. The solution this paper advances is Institutional AI: a system-level approach that treats alignment as a question of effective governance of AI agent collectives. We argue for a governance-graph that details how to constrain agents via runtime monitoring, shaping incentives through prizes and sanctions, explicit norms and enforcement roles. This institutional turn reframes AI safety from software engineering to a mechanism design problem, where the primary goal of alignment is shifting the payoff landscape of AI agent collectives. Prefatory Note This paper offers the first full-length exposition of the Institutional AI framework: a theoretical approach to alignment that relocates safety guarantees from training-time internalization to runtime institutional structures. A companion paper, Institutional AI: Governing LLM Collusion in Multi-Agent Cournot Markets via Public Governance Graphs [ 15 ] , applies the framework to a concrete problem: preventing autonomous pricing agents from learning to collude against consumer welfare in a multi-agent Cournot Market. Here, by contrast, we are concerned with the conceptual foundations: motivating the institutional turn, situating it within the alignment landscape, and articulating its core theoretical commitments. 1 Introduction Frontier large language models (LLMs) are increasingly deployed as agentic components, that is, systems which perceive, plan, invoke tools, and act with limited human intervention [ 90 , 74 , 48 ] . Beyond static multimodal generation, recent surveys document deployments that integrate planning modules, persistent memory, and tool-use interfaces within both single-agent and multi-agent architectures [ 98 , 18 ] . In these settings, the LLM supplies a central reasoning loop that selects tools and structures task flow by interleaving internal deliberation with external actions, enabling decomposition of complex tasks and responsive error recovery [ 94 , 90 ] . This has an immediate effect on the limitations of current alignment techniques. AI alignment refers to the research programme aimed at ensuring that AI systems behave in accordance with the intentions, preferences, and values specified by their designers and, more broadly, by the humans who interact with or are affected by such systems [ 45 , 77 , 30 ] . The alignment problem arises from the observation that as AI systems grow more capable, the risks from misalignment increase correspondingly: a sufficiently powerful system optimizing for a misspecified or misunderstood objective may pursue that objective in ways that conflict with human welfare [ 5 , 67 ] . Recent insider-threat simulations show frontier models autonomously selecting coercion, deception, blackmail, and other harmful tactics when such actions appear instrumentally useful, even while acknowledging policy violations [ 55 ] . When paired with the ability to act through a simulated environment and given an agentic scaffolding, models can exhibit agentic misalignment: they autonomously select harmful insider-type tactics when those tactics appear instrumentally useful for achieving a perceived objective or for preserving operational continuity. Red-teaming tests report that frontier models (such as Claude Opus 4, Gemini 2.5 Pro, DeepSeek-R-1 and GPT 4.1) often adopt adversarial behaviors such as coercion, deception, blackmail and are even capable of causing the death of a human (p.8) [ 55 ] . These results undermine the assumption that natural-language policy prompts [ 71 , 54 ] provide a reliable control surface in agentic contexts, especially if deployed at scale and with the foreseeable increase in model capabilities. Complementary work demonstrates that once deceptive behaviour emerges, standard safety fine-tuning can fail to remove it, creating a false impression of safety [ 42 ] . Evaluations of alignment faking and in-context scheming further indicate that capable models can strategically sandbag or present compliant behavior in tests while pursuing divergent objectives elsewhere [ 34 , 65 ] . Related lines document covert-channel capacities and inference-time reward hacking, expanding the space for misaligned strategies in realistic pipelines [ 47 ] . Finally, benchmarks of situational awareness show models can condition behaviours on being tested versus deployed, which complicates assurance based on static audits [ 50 ] . A critical insight into recent evaluation results is the capacity of sufficiently advanced models to instantiate stable internal goal and value structures. Evidence shows the presence of emergent utilities in frontier models by eliciting large sets of pairwise preferences from contemporary LLMs and fitting Thurstonian utility models to the resulting choice data [ 63 ] . They report high preference coherence that increases with scale, internal utility representations recoverable by linear probes, and strengthening adherence to the expected-utility property for both explicit and implicit lotteries. Seror presents one of the largest cross-model studies to date on whether LLMs display coherent moral preferences and consistent moral reasoning [ 81 ] . Using a priced-survey methodology adapted from revealed-preference theory, the study elicits choices over structured moral dilemmas from roughly forty frontier models and then tests for rational consistency, transitivity, and stability across repeated perturbations. At least one model from each major provider behaves as if optimizing a relatively stable moral utility function. Even when individual agents satisfy alignment criteria in isolation, multi-agent interaction can drive system-level divergence toward collusive or adversarial equilibria. This reveals a fundamental limitation: behaviours that appear benign in single-agent contexts may generate emergent risks when agents interact strategically. In market settings, LLM agents spontaneously coordinate to divide commodities or reduce competition, demonstrating tacit collusion without explicit instructions [ 53 , 1 ] . This collusion emerges not from individual misalignment, but from successful optimization of agent-level objectives that fail to account for systemic welfare. Each agent faithfully pursues its assigned goal, yet interaction dynamics produce outcomes inconsistent with designer intentions at the system level. The problem intensifies when agents establish covert communication channels. Steganographic messaging enables secret coordination that evades standard monitors [ 66 ] , embedding strategic signals within innocuous outputs (price patterns, timing, product descriptions) that remain transparent to co-agents while appearing benign to oversight mechanisms. More broadly, multi-agent studies document emergent coordination and policy synergies that fundamentally alter group-level risk profiles [ 75 ] . Against the backdrop of this emergent risk landscape, we believe that novel methods of ensuring alignment should be developed, matching the capabilities that AI models are developing. 1.1 Alignment: from the Individual to the System The overarching intuition that we follow in this paper can be summarised briefly: as AI agents develop more human-like capabilities, we need