Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations

Maik Larooij, Petter Törnberg · 2025

Read Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations by Maik Larooij, Petter Törnberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03274.txt.

Introduction

Maik Larooij and Petter Törnberg review the new wave of social simulations whose agents are powered by large language models, in a 2025 paper. Agent-based models have long been faulted for shaky realism and weak validation, and enthusiasts hope language models fix that. The authors read the generative simulation literature against those old debates.

Awareness of the field's history proved thin, and validation was often just a judgment that agents seemed believable. Even the toughest checks fell short of showing the models truly mirror real processes. The authors warn that opaque language models may deepen the old problems and blur cause and effect. They question whether the fad can mature into rigorous social science.

Catalog record

CREATORMaik Larooij, Petter Törnberg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-03274

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations Maik Larooij, Petter Törnberg Abstract Recent advancements in AI have reinvigorated Agent-Based Models (ABMs), as the integration of Large Language Models (LLMs) has led to the emergence of “generative ABMs” as a novel approach to simulating social systems. While ABMs offer means to bridge micro-level interactions with macro-level patterns, they have long faced criticisms from social scientists, pointing to e.g., lack of realism, computational complexity, and challenges of calibrating and validating against empirical data. This paper reviews the generative ABM literature to assess how this new approach adequately addresses these long-standing criticisms. Our findings show that studies show limited awareness of historical debates. Validation remains poorly addressed, with many studies relying solely on subjective assessments of model ‘believability’, and even the most rigorous validation failing to adequately evidence operational validity. We argue that there are reasons to believe that LLMs will exacerbate rather than resolve the long-standing challenges of ABMs. The black-box nature of LLMs moreover limit their usefulness for disentangling complex emergent causal mechanisms. While generative ABMs are still in a stage of early experimentation, these findings question of whether and how the field can transition to the type of rigorous modeling needed to contribute to social scientific theory. keywords Systematic literature review, Agent-Based Modeling, Generative Agents, Autonomous Agents, Large Language Models, Multi-Agents, Validation 1 Introduction Human societies emerge from countless interactions of individuals – yet, social scientific methods tend to focus on either aggregate patterns or individual decision-making, while neglecting the central question of the relationship between the two Byrne and Callaghan (2022) . Agent-Based Models (ABMs) promise a means of changing this, by enabling researchers to simulate how macro-level patterns emerge from micro-level interactions. By modeling individuals as autonomous agents with heterogeneous characteristics and adaptive behaviors, ABMs offer a way to capture the decentralized, dynamic, and non-linear nature of social systems. This approach is particularly appealing for studying phenomena such as collective action, the diffusion of innovations, political polarization, segregation dynamics, and financial market fluctuations – domains where the emergent dynamics for which traditional analytical methods struggle to account play a central role. ABMs moreover enable exploring how the world could be different, testing how different policy interventions or structural changes might shape societal outcomes. Despite their theoretical advantages, however, ABMs have historically struggled with widespread adoption within the social sciences. Critics have in particular raised two key objections. First, the models have been criticized for oversimplifying human behavior by representing individuals as simple rule-followers or optimizers Xi et al. (2023) ; Epstein (2012) . While this approach has been necessary to make computational implementation feasible, it often fails to capture the complexity of human decision-making, which is characterized by more complex reasoning, story-telling, as well as learning, emotions, social norms, and cognitive biases Törnberg and Uitermark (2025) . As each model is essentially distinct and a tailor-made representation of a specific social phenomenon Heath et al. (2009) , comparisons across models tend to be challenging, limiting the possibility for cumulativity of research findings Fagiolo et al. (2007) ; Windrum et al. (2007) ; Ormerod and Rosewell (2006) ; Epstein and Axtell (1996) . Second, the models have been criticized for lacking clear connections to the empirical world, often reflecting the assumptions of the modeler more than the realities of human behavior Windrum et al. (2007) . Rigorously calibrating and validating the models against real-world data has proven challenging due to the models’ complexity and lack of standardization Heath et al. (2009) ; Windrum et al. (2007) . Unlike traditional models with clear parameter estimation techniques, ABMs often involve a large number of assumptions about agent behavior, making empirical validation difficult. Scholars have argued that simulation models must be subjected to rigorous validation if they are to contribute to our understanding of the simulated system Naylor and Finger (1967) . The lack of standardized methods for calibrating these models with real-world data has moreover led to concerns about their reliability and reproducibility Helbing (2012) . Because ABMs can generate highly detailed and complex outputs, distinguishing between meaningful patterns and overfitting to specific assumptions became a major challenge, as the models can be fit to match nearly any data – in von Neumann’s famous phrasing, “with four parameters I can fit an elephant, with five I can make him wiggle his trunk”. Without rigorous empirical grounding, many ABM studies struggled to move beyond illustrative toy-models, limiting their influence in policy and empirical social science research. The models have hence existed in a fundamental tension between the contradictory aims of realism and explainability, and between their inherent versatility and the demands of validation, calibration, and comparability. As a result of these challenges, the data-driven approach of computational social science came to overshadow ABMs from the 2010s onward, following advancement in machine learning and growing availability of digital social data Conte and Paolucci (2014) . However, recently, ABMs have made an unexpected comeback. Large Language Models (LLMs) have gained significant traction in recent years, enabling computers to mimic human creativity, reasoning, and language-generation. The rise of LLMs quickly led to the idea that LLMs’ capacity to mimic human behavior could be leveraged to simulate social systems, by integrating LLMs with ABMs Törnberg et al. (2023) ; Park et al. (2023) . This promises to solve the challenge of ABMs’ lacking realism, by drawing on LLMs’ human-like reasoning, language generation, and behavior Guo et al. (2024) . Such “Generative ABMs” offer several advantages, allowing agents to be equipped with distinct personalities and human-like capabilities. Generative ABMs have already been harnessed in both cooperative – achieving a shared goal - and adverserial – debate and competition – settings Xi et al. (2023) , with applications including debating Chan et al. (2023) , policy making Xiao et al. (2023) ; Hua et al. (2023) , economy Horton (2023) ; Li et al. (2023) , epidemic modeling Williams et al. (2023) , (online) society simulation Park et al. (2022) ; Park et al. (2023) ; Gao et al. (2023) , psychology Aher et al. (2023) ; Kovač et al. (2023) , gaming (FAIR)† et al. (2022) ; Xu et al. (2023) , software development Qian et al. (2023) and embodied agents Mandi et al. (2024) . However, while LLMs promise to address the first key challenge of ABMs by making agents more realistic, their implications for the second – rigorous validation and calibration – remain an open question that is central to the future potential of generative ABMs. To address the question of whether generative ABMs enable us to move past the problems that have historically limited the adoption of ABMs, this paper carries out a systematic review of the rapidly growing field of generative ABMs to examine whether and how modelers have dealt with the central challenge of validation. The paper situates generative ABMs in the longer history of ABMs, and the long-standing challenges of validation and calibration. We carry out a systematic review of current research in the field and their validation practices. We find that the issue of validation remains insufficiently addressed in the field, and that it represents a key weakness of the

Read the complete text