Validating Generative Agent-Based Models of Social Norm Enforcement: From Replication to Novel Predictions
Logan Cross, Nick Haber, Daniel L.K. Yamins · 2025
Read Validating Generative Agent-Based Models of Social Norm Enforcement: From Replication to Novel Predictions by Logan Cross, Nick Haber, Daniel L.K. Yamins (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22049.txt.
Introduction
Logan Cross, Nick Haber, and Daniel Yamins test AI social simulations in a 2025 paper. Language model agents can now act out human social scenes, but trust in their behavior needs proof. The team validate their setup in two stages using classic social dilemma studies. First they find which mental parts agents need to copy known human results.
Both persona based differences and theory of mind proved essential for matching third party punishment, where bystanders punish unfair acts. With that architecture locked, the team ran fresh conditions the human studies never tried. The recipe offers a path from mere imitation to prediction, the step that would make agent simulations a real research tool.
Catalog record
| CREATOR | Logan Cross, Nick Haber, Daniel L.K. Yamins |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22049
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Validating Generative Agent-Based Models of Social Norm Enforcement: From Replication to Novel Predictions Logan Cross, Nick Haber, Daniel L.K. Yamins Abstract As large language models (LLMs) advance, there is growing interest in using them to simulate human social behavior through generative agent-based modeling (GABM). However, validating these models remains a key challenge. We present a systematic two-stage validation approach using social dilemma paradigms from psychological literature, first identifying the cognitive components necessary for LLM agents to reproduce known human behaviors in mixed-motive settings from two landmark papers, then using the validated architecture to simulate novel conditions. Our model comparison of different cognitive architectures shows that both persona-based individual differences and theory of mind capabilities are essential for replicating third-party punishment (TPP) as a costly signal of trustworthiness. For the second study on public goods games, this architecture is able to replicate an increase in cooperation from the spread of reputational information through gossip. However, an additional strategic component is necessary to replicate the additional boost in cooperation rates in the condition that allows both ostracism and gossip. We then test novel predictions for each paper with our validated generative agents. We find that TPP rates significantly drop in settings where punishment is anonymous, yet a substantial amount of TPP persists, suggesting that both reputational and intrinsic moral motivations play a role in this behavior. For the second paper, we introduce a novel intervention and see that open discussion periods before rounds of the public goods game further increase contributions, allowing groups to develop social norms for cooperation. This work provides a framework for validating generative agent models while demonstrating their potential to generate novel and testable insights into human social behavior. Introduction Advances in LLMs have created new possibilities for studying human social behavior through simulation. While agent-based modeling has long been used in the social sciences, the emergence of LLM-based agents enables a qualitatively new approach to simulate humans: generative agent-based modeling (GABM) ( Vezhnevets et al. (2023) ; Anthis et al. (2025) ) . Generative agents are LLM-powered entities that combine multiple cognitive components—such as memory, personality, and reasoning modules—through structured prompt sequences to simulate human decision-making processes ( Park et al. (2023) ; Park et al. (2024) ) . These agent architectures function as computational models of human cognition, similar to the ”box-and-arrow” diagrams used in cognitive psychology ( Neisser (2014) ) . For GABMs to be useful tools for social science, we need rigorous frameworks to validate whether these agents actually capture relevant aspects of human behavior. By presenting agents with the same experimental stimuli used in human studies, we can directly test whether their stimulus-to-behavior mappings match those observed in human participants. Thus, in this paper we introduce a two-stage approach that prototypes GABMs to replicate and extend human studies. 1 1 1 Code: https://github.com/locross93/Multi_Agent Model Validation: First, we conduct an iterative model comparison to identify the minimal agent architecture (set of cognitive components) sufficient to reproduce known behavioral effects in complex social settings. Each “model” in our framework represents a different LLM agent architecture—specifically, different combinations of prompts which we call cognitive components. These components are added to a base agent who takes in the responses to these prompts as context for making a decision, for example the answer to “what kind of person am I?” provides additional context to guide the agents decisions about which actions to take ( March & Olsen (2008) ; Vezhnevets et al. (2023) ) . Our validation of each model works as follows: We run the same experimental protocol used in the original human studies by representing the experimental stimuli in natural language and compare our agents’ behavioral patterns to the published human data using identical statistical tests. An architecture is considered “validated” when agents show the same directional effects as humans with statistical significance. For theoretical parsimony in line with suggested best practices for GABM ( Vezhnevets et al. (2023) ) , we begin with the simplest possible agent architecture and test new cognitive components only when simpler models fail to capture human behavior patterns. In addition, by varying the agent architectures and comparing their output to human data, we can test specific hypotheses about the cognitive mechanisms underlying social behavior. Novel Prediction Generation: Second, we leverage validated agent architectures to explore counterfactual scenarios that would be slow or costly to implement in laboratory settings. By maintaining the validated agent architectures while varying environmental conditions, we can generate precise, quantitative predictions about human behavior in novel situations. These predictions can then be tested empirically, providing a rigorous test of the model’s generalization capabilities while advancing our understanding of human social behavior. Here, we present this validation approach with two case studies to examine the components necessary for GABMs to replicate key findings from experimental social dilemmas. Mixed-motive social dilemmas, particularly the trust game and public goods games, represent the gold standard for experimental research on social dynamics across psychology, economics, and social sciences because they uniquely capture the fundamental tension between individual self-interest and collective welfare that defines human social life ( Van Lange et al. (2013) ) . Unlike approaches that study cooperation or competition in isolation, these paradigms model the complex mixed incentives that characterize most important social challenges — from maintaining personal relationships ( Crocker et al. (2017) ) to climate change cooperation ( Milinski et al. (2008) ) to following cultural norms ( Köster et al. (2022) ; Leibo et al. (2024) ) . Greater theoretical frameworks of social dilemmas help us to understand why people cooperate (or not) and also practical ways to maintain and promote cooperation in groups and organizations through coordination and social norm enforcement. This research has revolutionized our understanding through methodologically rigorous frameworks that bridge laboratory precision with real-world relevance ( Fehr & Gächter (2000) ; Henrich et al. (2006) ) . First, we investigate third-party punishment (TPP) in the trust game from Jordan et al. (2016) , which demonstrated that third-party punishers are perceived as—and actually are—more trustworthy than non-punishers. Through careful comparisons of different agent architectures built in Concordia, we show that both persona prompting and theory of mind reflection are necessary components for reproducing these effects. Second, we examine Feinberg et al. (2014) ’s work on gossip and ostracism in promoting cooperation in a public goods game, finding that while our initial validated architecture from the TPP study provides a good foundation, an additional strategic component proves necessary to replicate a majority of the behavioral effects in this more complex social environment. We then apply these validated model architectures to explore two distinct types of novel questions not investigated in the original papers. For the TPP study, we use the validated model for theoretical disambiguation, examining how punishment behavior differs between public and private settings. Our results quantify the relative contributions of reputational vs. intrinsic motivations, helping resolve the ongoing theoretical debate about why people engage in costly third-party punishment. For the public goods study, we use the validated model to test a novel intervention—adding pre-round discussion periods—and find that this further improved cooperation rates, as groups could self-organize and establish shared norms for contribution. These findings demonstrate how GABM can help social scientists both