Position: LLM Social Simulations Are a Promising Research Method
Jacy R. Anthis, Ryan Liu, Sean M. Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, Michael S. Bernstein · 2025
Read Position: LLM Social Simulations Are a Promising Research Method by Jacy R. Anthis, Ryan Liu, Sean M. Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, Michael S. Bernstein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02234.txt.
Introduction
Jacy Anthis and co-authors argued in 2025 that simulating human subjects with language models is a promising research method. Accurate, checkable simulations would give social science a cheap, open data source. Early attempts, though, were limited, and few social scientists have taken the method up.
The authors ground their case in comparisons of model output against real human subjects and in expert commentary. Five fixable challenges stand between the idea and its promise. Rich context in prompts and fine-tuning on social science data are among the promising fixes. Simulations already suit pilot and exploratory work, they argue. Better conceptual models and repeated checks are the path to broader trust.
Catalog record
| CREATOR | Jacy R. Anthis, Ryan Liu, Sean M. Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, Michael S. Bernstein |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-02234
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position: LLM Social Simulations Are a Promising Research Method Jacy R. Anthis, Ryan Liu, Sean M. Richardson, Austin C. Kozlowski, Bernard Koch, Erik Brynjolfsson, James Evans, Michael S. Bernstein Abstract Accurate and verifiable large language model (LLM) simulations of human research subjects promise an accessible data source for understanding human behavior and training new AI systems. However, results to date have been limited, and few social scientists have adopted this method. In this position paper, we argue that the promise of LLM social simulations can be achieved by addressing five tractable challenges. We ground our argument in a review of empirical comparisons between LLMs and human research subjects, commentaries on the topic, and related work. We identify promising directions, including context-rich prompting and fine-tuning with social science datasets. We believe that LLM social simulations can already be used for pilot and exploratory studies, and more widespread use may soon be possible with rapidly advancing LLM capabilities. Researchers should prioritize developing conceptual models and iterative evaluations to make the best use of new AI systems. 1 Introduction With the quickly increasing humanlikeness of large language models (LLMs), many researchers are investigating their use for simulating human research subjects. This could address many limitations of human data, including difficulties of representative sampling ( Henrich et al. 2010 ) , financial costs that limit accessibility ( Alemayehu et al. 2018 ) , and methodological biases such as non-response bias ( Sedgwick 2014 ) . Complementing human data with humanlike simulations could accelerate social science, open up new research opportunities—such as exploring historical or potential future counterfactuals and piloting large-scale policy changes—and provide high-quality synthetic data for the development of human-centered AI at scale ( Bai et al. 2022 ; Kim et al. 2023 ) . Nonetheless, the limitations of LLMs and simulation results to date have cast doubt on whether accurate and verifiable simulation is possible ( Agnew et al. 2024 ; Gao et al. 2024 ; Wang et al. 2024a ; Wang et al. 2024b ) . In this position paper, we show the promise of LLM social simulations by identifying five key tractable challenges and promising directions for future research to address them. We summarize the challenges in Table 1 : diversity, bias, sycophancy, alienness, and generalization. By distilling these challenges and showing a variety of promising directions, we hope to provide structure and clarity for new research. Our argument is grounded in a literature review of empirical studies that have compared human research subjects to LLMs, commentaries on the topic, and related work in social science and other LLM applications. Compelling simulation results so far include: • Hewitt et al. 2024 , the largest test of sims to date, spanned 70 preregistered and U.S.-representative experiments alongside an archive of replication studies. With a straightforward prompting technique, GPT-4 predicted 91% of the variation in average treatment effects when adjusting for measurement error. • Binz et al. 2024 fine-tuned Llama-3.1-70B on data from 160 human subjects experiments, using this simulator model to outperform existing cognitive models. • Park et al. 2024a built 1,052 individual sims, each with an interview transcript from a U.S.-representative sample. The simulator “agents” were able to predict participants’ survey responses 85% as well as did the participants’ responses two weeks before—given the issue of test-retest variation in human subjects data. Table 1: LLM social simulations must address five key challenges. Challenge Description Promising Directions Diversity Generic and stereotypical outputs that lack human diversity Inject humanlike variation in training, tuning, or inference (e.g., interview-based prompting, steering vectors) Bias Systematic inaccuracies when simulating particular human groups Prompt with implicit demographic information; minimize accuracy-decreasing biases rather than all social biases Sycophancy Inaccuracies due to excessively user-pleasing outputs Reduce the influence of instruction-tuning; instruct LLM to predict as an expert rather than roleplay a persona Alienness Superficially accurate results generated by non-humanlike mechanisms Simulate latent features; iteratively conceptualize and evaluate; reassess as mechanistic interpretability advances Generalization Inaccuracies in out-of-distribution contexts, limiting scientific discovery Simulate latent features; iteratively conceptualize and evaluate; reassess as generalization capabilities advance Most studies have used only a small fraction of the methods that can increase simulation accuracy, leaving substantial room for improvement. Evidence from simulation studies is bolstered by broader evidence of LLM capabilities as they have saturated existing benchmarks ( Maslej et al. 2025 ) , leading to efforts towards an “evaluation science” ( Weidinger et al. 2025 ) , and rapid growth in more realistic measures, such as the length of tasks that AI can do ( METR 2025 ) . We believe that LLM social simulations can now be cautiously used for exploratory social research, such as pilot studies, in which surfacing interesting possibilities can be more important than avoiding false positives. Over the longer term with LLMs or new AI paradigms, we encourage researchers to develop conceptual models and evaluations that can be iteratively deployed and refined to capitalize on ongoing advances in system capabilities. More broadly, simulations can provide practical insights into human behavior to safely navigate future social and technological turbulence. Conceptual insights can identify what is required for an AI system to be meaningfully humanlike, helping us address profound challenges as humans come to coexist with “digital minds” ( Anthis et al. 2025 ) . 2 Scope This paper builds an agenda for LLM social simulations (shortened to sims ), which we define as the use of language modeling to generate accurate and verifiable data that can be used as if it were behavioral data collected from human research subjects. These are sometimes called social simulation “agents” ( Park et al. 2024a ) , though we do not restrict our scope to simulations that have agency ( Kenton et al. 2022 ) . LLM social simulations overlap with several closely related research areas, including the use of social science methods to understand LLMs ( Kosinski 2024 , e.g.,) ; comparisons between LLMs and humans to understand LLMs ( Zhang et al. 2025a , e.g.,) ; the study of LLM roleplay ( Chen et al. 2024 , e.g.,) ; non-LLM simulations of human research subjects ( Romero et al. 2023 , e.g.,) ; and the use of LLMs for research tasks other than social simulations, such as annotation ( Aubin Le Quéré et al. 2024 , e.g.,) and conducting research ( Si et al. 2024 , e.g.,) . We summarize the studies in our primary scope in Table A1 and selected other work in Table A2 , preprinted or published by May 31, 2025. We discuss details of the reviewed studies in Appendix A . Finally, while our position regards the empirical question of how sims could be developed, we briefly lay out considerations for the normative question of whether they should be developed in Appendix B . 3 Challenges Because of the overlapping nature of the five challenges, we enumerate all five before proposing future directions. 3.1 Diversity In our vision of accurate and verifiable LLM social simulations, a central requirement is diversity , the extent to which a simulation matches the variation of the human population being simulated. Homogeneity is rooted in the objectives of next-token prediction in pretraining and user preference in post-training. The lack of diversity has been