Many LLMs Are More Utilitarian Than One

Anita Keshmirian, Razan Baltaji, Babak Hemmatian, Hadi Asghari, Lav R. Varshney · 2025

Read Many LLMs Are More Utilitarian Than One by Anita Keshmirian, Razan Baltaji, Babak Hemmatian, Hadi Asghari, Lav R. Varshney (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00814.txt.

Introduction

Anita Keshmirian and co-authors find language models, like humans, grow more willing to harm one for the greater good when deliberating in groups, in a 2025 paper. Human groups show a utilitarian boost, endorsing norm violations that maximize benefit. The team tested six models on classic moral dilemmas, alone and in pairs or triads holding multi-turn discussions.

In personal dilemmas, where the agent would directly harm someone, every model rated the violation more acceptable after group discussion. The boost matched the human pattern in direction, not mechanism. Human groups grow outcome-sensitive; model groups cared less about norms or weighed everyone's interests more impartially. Models differed in when and how strongly the boost appeared; prompt and group composition could strengthen or soften it. The authors draw lessons for alignment and for multi-agent systems whose collective morals drift from their members'.

Catalog record

CREATORAnita Keshmirian, Razan Baltaji, Babak Hemmatian, Hadi Asghari, Lav R. Varshney
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE101 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00814

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Many LLMs Are More Utilitarian Than One Anita Keshmirian, Razan Baltaji, Babak Hemmatian, Hadi Asghari, Lav R. Varshney Abstract Moral judgment is integral to large language model (LLM) alignment and social reasoning. As multi-agent systems gain prominence, it becomes crucial to understand how LLMs function collectively during collaboration, compared to individual agents. In human moral judgment, group deliberation leads to a utilitarian boost: a tendency to endorse norm violations that maximize benefits for the greatest number of people despite harms. We study whether a similar dynamic emerges in multi-agent LLM systems. We tested six models on well-established sets of moral dilemmas across two conditions: (1) Solo, where models reasoned independently, and (2) Group, where they engaged in multi-turn discussions in pairs or triads. In personal moral dilemmas, where agents must decide to directly harm one individual to maximize the utility for others, all models found moral violations to be more acceptable when part of a group than individually, similar to human experiments. Some models endorsed actions that maximized overall well-being, even if they benefited strangers over familiar individuals. Others became more willing to violate moral norms in groups. However, while human groups show a similar action bias, the mechanism for their utilitarian boost differs from LLMs. Whereas the human shift comes from heightened sensitivity to decision outcomes, LLM groups show either reduced norm sensitivity or enhanced impartiality. This suggests that while the surface behavior of LLM collectives mimics human group reasoning, the underlying drivers differ. We discuss the implications for AI alignment, multi-agent design, and artificial moral reasoning. † † footnotetext: [email protected] , [email protected] 1 Introduction Multi-agent systems (MAS) provide a key paradigm for analyzing decentralized decision-making and coordination among multiple autonomous entities in dynamic environments. By incorporating large language models (LLMs), these systems, referred to as LLM-MAS, feature agents with enhanced capabilities in reasoning, communication, and knowledge access [ 1 ] . This integration significantly expands the scope and complexity of tasks MAS can tackle, supporting more adaptive, flexible, and human-aligned agent behaviors [ 2 ] . However, integrating LLMs into MAS also introduces a new class of risks. When agents negotiate through iterative message passing or refine solutions jointly, even minor alignment discrepancies can cascade into major emergent distortions. Bias amplification [ 3 ] , covert coordination protocols [ 4 ] , knowledge drift [ 5 ] , conflicting agreements [ 6 ] , and collusion [ 7 , 8 ] may emerge, all of which remain invisible when each LLM agent is evaluated in isolation. As multiple LLM-based assistants collaborate to tackle increasingly complex tasks, understanding how collective reasoning emerges from individual models is imperative for ensuring safety and trustworthiness [ 9 ] . Figure 1: A schematic representing our experimental setup for LLM moral deliberation and reflection. A triad of of LLM agents engage in multi-round discussions about moral dilemmas and conclude with private reflections. This example illustrates how the group setting induces a utilitarian boost even when such actions involve moral norm violations. Recent work has begun to draw on insights from social psychology to investigate emergent distortions in group-level multi-agent LLM reasoning, demonstrating phenomena such as conformity [ 10 , 11 ] , belief congruence [ 12 ] , and social irrationality [ 4 ] . Although not unique to LLM-MAS, these emergent patterns often show both similarities to and differences from well-studied collective behaviors in human groups [ 13 ] . Such collective distortions become especially important in moral domains where real-world consequences of norm violations can be disastrous, as in healthcare, education, and law. Aligning individual models will be insufficient to guarantee aligned group outcomes in such cases, and debiasing would need to take into account the collective element of the decision-making. But past research on LLM moral reasoning has overwhelmingly focused on comparing isolated agents with humans, as in the Moral Machine Experiment and the Moral Reasoning Benchmarks [ 14 , 15 , 16 ] . Thus, it provides little guidance for the emergent collective moral dynamics. This knowledge gap increases the risk of missing important deviations from normative guidelines, with strong possible real-world consequences. Moral decision-making is increasingly delegated to LLM collectives across high-stakes domains—for instance, coordinating specialized doctor and patient agents in simulated clinical settings [ 17 ] , and decomposing complex legal tasks for improving legal reasoning and rule interpretation [ 18 ] . In these settings, covert consensus protocols could lead to collective LLM decisions that contradict those of reasonable humans, any normative frameworks, and even the output of any single model. This creates an urgent blind spot: without analyzing group-level moral dynamics in LLM-MAS, we cannot understand, predict, or prevent ethically problematic outcomes that escape single-agent safety evaluations. Suppose, for instance, that clinical LLM groups exhibit an emergent willingness to sacrifice individuals for “the greater good” in contrast with a physician’s Hippocratic oath. The resulting loss of trust in experts and decision-making institutions could be potentially catastrophic. Studying group dynamics in moral LLM decision-making also fills an important gap in computational social science, where the potential of LLMs for simulating human reasoning or determining whether human tendencies arise from language use is receiving greater attention [ 19 ] . Ignoring the deeply social nature of moral reasoning—where social deliberation, argumentation, and negotiation play a central role—also limits the insights afforded by simulation experiments. This study applies controlled experiments and validated measurement tools from psychology to address the discussed gaps by systematically investigating how collective moral reasoning evolves in LLM-MAS. In doing so, it focuses on three central questions: 1. Are LLMs more likely to violate moral norms in the service of “the greater good” when deliberating in small groups? We operationalize this “utilitarian boost” by placing multiple copies of the same LLM in a structured dialogue, asking them to reach consensus on diverse sets of well-established moral scenarios. 2. How do multi-agent LLM judgments compare to human group judgments? Drawing on established human data on group deliberation studies, we examine both quantitative shifts (e.g., change in utilitarian endorsement rates) and the possible mechanisms at play. 3. How do any shifts or differences arise? We compare theories from moral psychology as explanations, seeking evidence for outcome-focused reasoning versus norm-violation rationalizations. We test multiple LLMs to assess mechanistic variability 1 1 1 Mechanism refers to patterns in LLMs’ outputs across conditions, not architectures or cognitive processes. As a preview, our experiments reveal a concerning phenomenon that has not been demonstrated in multi-agent LLM systems before: when identical agents engage in collective deliberation, they do not merely aggregate their individual moral judgments. Instead, they exhibit a consistent utilitarian boost—a systematic shift toward endorsing actions that maximize overall welfare, even when such actions involve sacrificing or harming a minority of humans. After six rounds of inter-agent exchange, groups reach a utilitarian decision that is more approving of moral norm violations than the corresponding single model. This utilitarian boost appears in six state-of-the-art LLMs and is therefore unlikely to reflect specific training datasets or paradigms. Reliability checks through human ratings make incoherent responses or misalignment between moral ratings and argumentation unlikely explanations for the change. While a similar utilitarian boost has been observed in human groups [ 20 , 21 ,

Read the complete text