The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions

Azza Bouleimen, Giordano De Marzo, Taehee Kim, Nicolò Pagan, Hannah Metzler, Silvia Giordano, Anikó Hannák, David Garcia · 2025

Read The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions by Azza Bouleimen, Giordano De Marzo, Taehee Kim, Nicolò Pagan, Hannah Metzler, Silvia Giordano, Anikó Hannák, David Garcia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08592.txt.

Introduction

Azza Bouleimen and seven co-authors test whether language models can fake a group conversation well enough to fool readers. They gathered real Reddit threads and generated artificial ones on the same topics with two models, Llama 3 and GPT-4o, then showed both kinds to study participants side by side.

Readers mistook the model-made threads for human writing thirty-nine percent of the time. For the Llama 3 threads, people spotted the machine authorship only fifty-six percent of the time, barely better than a coin flip. The authors read the result two ways: simulated communities could help test platform policies and recommendation designs, but the same realism could flood social media with fake discussion.

Catalog record

CREATORAzza Bouleimen, Giordano De Marzo, Taehee Kim, Nicolò Pagan, Hannah Metzler, Silvia Giordano, Anikó Hannák, David Garcia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08592

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Collective Turing Test: Large Language Models Can Generate Realistic Multi-User Discussions Azza Bouleimen, Giordano De Marzo, Taehee Kim, Nicolò Pagan, Hannah Metzler, Silvia Giordano, Anikó Hannák, David Garcia Abstract Large Language Models (LLMs) offer new avenues to simulate online communities and social media. Potential applications range from testing the design of content recommendation algorithms to estimating the effects of content policies and interventions. However, the validity of using LLMs to simulate conversations between various users remains largely untested. We evaluated whether LLMs can convincingly mimic human group conversations on social media. We collected authentic human conversations from Reddit and generated artificial conversations on the same topic with two LLMs: Llama 3 70B and GPT-4o. When presented side-by-side to study participants, LLM-generated conversations were mistaken for human-created content 39% of the time. In particular, when evaluating conversations generated by Llama 3, participants correctly identified them as AI-generated only 56% of the time, barely better than random chance. Our study demonstrates that LLMs can generate social media conversations sufficiently realistic to deceive humans when reading them, highlighting both a promising potential for social simulation and a warning message about the potential misuse of LLMs to generate new inauthentic social media content. keywords Large Language Models, Social Media Moderation, Turing Test, Social Simulation, Online Discussions, Reddit Introduction Agent-Based Modeling (ABM) approaches have been used to identify principles that drive the emergence of collective behavior such as political polarization [ 1 ] , segregation [ 2 ] , collective emotions [ 3 ] cultural dynamics [ 4 ] and collective action [ 5 ] . While ABM can be used to illustrate generative sufficiency [ 6 ] , i.e. the minimal conditions for a social phenomenon to emerge, they tend to oversimplify human behavior [ 7 ] , leading to limited application in policies or interventions. To increase the practical applicability of ABM in policy-making or system design, the dynamics of individual behavior and interactions between agents need to be grounded on empirical data [ 8 ] . However, data scarcity and the variance of human behavior complicates the task of model calibration and limits the ability of ABM simulations to generate predictions. Recent advances in generative AI, in particular Large Language Models (LLMs), promise to narrow this gap as they demonstrate high role-play capabilities [ 9 , 10 , 11 , 12 ] . LLMs can enable a new kind of support for the study and simulation of collective behavior [ 13 , 14 ] by complementing empirical data and allowing agents to communicate through real text or to decide agent actions based on LLM inference [ 15 ] . Social media are gaining significant traction as a prime application of social simulation. In light of the harm spreading on social media platforms [ 16 , 17 ] and their alarming real-world consequences [ 18 , 19 , 20 , 21 , 22 , 23 ] , research on interventions and policies is needed to understand how to moderate the conversation and information flow on social media platforms. However, some intervention techniques [ 24 ] fail to mitigate online social issues [ 25 , 26 ] , and can sometimes even cause further harm themselves [ 27 , 28 , 29 , 30 ] . Given the ethical concerns of conducting large-scale experimentation on social media [ 31 ] , generative ABM, i.e. the usage of generative agents in simulations, presents itself as an ideal safe and controlled environment where researchers and policymakers can design and test different intervention strategies. Simulating social media through generative ABM has become even more pressing in recent years in light of the restrictions to public data access that social media platforms have introduced [ 32 ] . Generative ABM has been previously used to simulate social media platforms and the interest they spark is on the rise. Early studies show that LLMs can link forming online social networks similar to those created by humans [ 33 ] . More recently, they have been employed to test how different feed algorithms affect the level of toxicity and cross-partisan interactions [ 34 , 35 , 36 ] . Moreover, several different frameworks for simulating social networks have been developed [ 36 , 37 , 38 ] . For example, recent work by Ferraro et al. [ 36 ] provides a comprehensive framework for building specific personas as agents in a simulated social media environment. Despite this growing interest, there are still many open questions connected to the use of GABM. One of the most pressing is the need for novel validation approaches that show whether generative agents represent social media users, react in ways comparable to humans, and create overall group behavior that is akin to what humans do on social media. In other words, we still need to investigate the extent to which LLMs can mimic human behavior on social media. Some studies have given initial insights into how conversations can be simulated with LLMs. Park et al. [ 39 ] introduced a prototyping technique to generate online forum discussions using GPT-3. For validation, they recruited crowdworkers to manually author comments on posts following authentic Reddit user comments. Both the crowdworker created conversations and the GPT-3 conversations were presented to a separate group of participants, who were asked to identify which conversation were AI-generated. The results showed that participants correctly identified AI-generated conversations 59% of the time, indicating that GPT-3 can be suitable for simulating social media-like conversations. We note, however, that the crowdworker-created conversations used for comparison in this experiment were not authentic Reddit conversations. This limits the external validity of the experiment, as the control condition of crowdworker conversations is likely not representative of true online interactions. To improve on this approach, we suggest in our paper an experiment based on genuine Reddit conversations, created by active Reddit users. Pagan et al. [ 40 ] present a computational framework to systematically evaluate differences between LLM-generated social media and human replies. Their results reveal persistent divergences between human and LLM-generated replies, especially in affective and stylistic dimensions. However, they also showcase clear gains in realism from stylistic conditioning, context-aware prompting, and fine-tuning. While their approach is purely analytical, relying on BERT models and other Machine Learning (ML) systems to distinguish between LLM and human replies, our study employs an experimental design with human participants. This approach is complementary to [ 40 ] , as insights provided by humans are likely to be different from what ML models detect. Besides the studies mentioned above, the overall reliability of generative ABM to simulate online conversations among various individuals remains under-explored. In a recent survey from Larooij and Törnberg [ 13 ] , the authors state that the “Validation of generative social simulations remain poorly addressed” and that “even the most rigorous validation fail to adequately evidence operational validity”. In this article, we present the results of a Collective Turing Test : Whether LLMs can generate the text of human-like conversations on social media. 75 years ago, Alan Turin described the Turing test to evaluate if a machine can generate a textual conversation that humans cannot distinguish from another human more than 30% of the time [ 41 ]

Read the complete text