Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations

Giacomo Fidone, Lucia Passaro, Riccardo Guidotti · 2025

Read Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations by Giacomo Fidone, Lucia Passaro, Riccardo Guidotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07204.txt.

Introduction

Giacomo Fidone, Lucia Passaro, and Riccardo Guidotti grade online content moderation by simulating social networks with language-model-driven agents. Whether moderation works is hard to measure: field data is costly and tests lack control. Their simulator runs parallel counterfactual worlds of the same online conversation, identical except that control actions act in one, letting toxic behavior spread or be contained under each strategy.

Extensive tests probe the psychological realism of the agents, reproduce social contagion phenomena, and compare control plans head to head. Personalized control came out superior to uniform methods. The tool fills a methods gap, giving researchers and platforms a controlled lab for testing control policies before deploying them on real communities, where mistakes carry social costs.

Catalog record

CREATORGiacomo Fidone, Lucia Passaro, Riccardo Guidotti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07204

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating Online Moderation Via LLM-Powered Counterfactual Simulations Giacomo Fidone, Lucia Passaro, Riccardo Guidotti Abstract Online Social Networks (OSNs) widely adopt content moderation to mitigate the spread of abusive and toxic discourse. Nonetheless, the real effectiveness of moderation interventions remains unclear due to the high cost of data collection and limited experimental control. The latest developments in Natural Language Processing pave the way for a new evaluation approach. Large Language Models (LLMs) can be successfully leveraged to enhance Agent-Based Modeling and simulate human-like social behavior with unprecedented degree of believability. Yet, existing tools do not support simulation-based evaluation of moderation strategies. We fill this gap by designing a LLM-powered simulator of OSN conversations enabling a parallel, counterfactual simulation where toxic behavior is influenced by moderation interventions, keeping all else equal. We conduct extensive experiments, unveiling the psychological realism of OSN agents, the emergence of social contagion phenomena and the superior effectiveness of personalized moderation strategies. This is a preprint version of the paper accepted for publication in the Proceedings of AAAI 2026. Code — https://github.com/gfidone/COSMOS Introduction Over the past two decades, Online Social Networks (OSNs) have witnessed the growing incidence of toxic behavior, encompassing “interactions designed to be inflammatory and purposefully breed counterproductive dissension” ( 26 ) . The spread of online toxicity has been magnified by the well-known online disinhibition effect ( 32 ) and a resulting affective polarization ( 56 ) , i.e., the tendency to develop hostile sentiments towards unlike-minded individuals, thus becoming a serious threat to the safety and mental health of OSN users ( 43 ) . This has urged social platforms to enforce moderation interventions , either ex post , aimed at punishing misbehaving users with ban and censorship; or ex ante , aimed at preventing recidivism through the delivery of text messages ( 23 ) . However, evaluating the effectiveness of moderation strategies is still challenging ( 12 ) . Gathering significant volumes of empirical evidence is hindered by the API restrictions imposed by private OSNs and the sparsity of toxic behavior itself. Also, field observation lacks full control over experimental variables, leaving no a priori assurance about the absence of potential, unknown confounders. Nevertheless, social sciences are undergoing a major methodological revolution, driven by the possibility to enhance Agent-Based Modeling (ABM) ( 40 ) with Large Language Models (LLMs) ( 6 ) for simulating human-like behavior across a wide range of social scenarios ( 52 ) . Hence, we argue that generating empirical evidence, rather than collecting it from the real world, can minimize costs while maximizing controllability. To this end, we introduce cosmos ( co unterfactual s imulations of mo deration s trategies ), a LLM-powered simulator of OSN conversations designed to support the evaluation of moderation strategies 1 1 1 Given the nature of the topic, we caution that the paper includes examples some may find offensive or disturbing. . cosmos implements LLM-based agents distinguished by different profiles and enabled to interact within a OSN-like environment. Unlike other tools, cosmos runs two parallel simulations: a factual simulation and a counterfactual simulation. The latter is a replica of the former, with all else kept constant except for the application of moderation interventions. As an example, Figure 1 presents a factual conversation generated by cosmos alongside its counterfactual version. In this way, cosmos allows to observe and measure how much a moderation strategy influences toxic behavior as it emerges from agents’ intrinsic dispositions and social interactions. We highlight that cosmos is designed to capture only the conversational dynamics of OSNs, thus excluding actions such as likes, follows and re-posts, which are less relevant to its objectives and would introduce additional variability. Hence, cosmos does not simulate social relationships. In this regard, the term “Online Social Network ” might be imprecise, but we retain it for consistency with related literature. Figure 1: Example of factual thread and its counterfactual version from cosmos experiments. In the counterfactual simulation, Agent 19 receives a moderation message at time 1 for having submitted a toxic post. The memory of this message influences Agent’s 19 behavior at subsequent timestamps. For example, at time 3 it is effective at mitigating the toxicity of Agent 19’s reply. In turn, this change has cascading effects on lower nodes: although Agent 6 has no memory of past moderation messages, at time 4 it reduces its toxicity. Some profile features of the two agents are displayed on the left (for full profiles, see Appendix A). Building on recent studies about content moderation ( 12 ) , we present a use case of cosmos to assess the advantages of personalized moderation. To do that, we implement two ex ante strategies: one-size-fits-all , where the moderation message is the same for all agents; and personalized moderation interventions, where moderation messages are tailored to the socio-psychological profile of each agent. Also, we implement a ban strategy to study the trade-off between mitigation and deplatforming effects. Key findings include: ( i ) the consistency and psychological believability of toxic behavior; ( ii ) the emergence of toxicity contagion phenomena; ( iii ) the superior effectiveness of personalized moderation. These results demonstrate how cosmos can be leveraged as a complement to field observation, both for research objectives, e.g., the validation of hypotheses in the social sciences, and industrial applications, e.g., the test of automated moderation systems. Related Works Our work intersects multiple research domains. Hence, we do not aim here to provide an exhaustive literature review. Instead, we highlight key studies that conceptually ground cosmos and outline main research directions. Finally, we position our proposal within this broad context. Socio-demographic and Psychological Prompting. Several studies have tested the ability of LLMs to understand and simulate human behavior from socio-demographic and psychological information ( 1 ; 51 ) . Notably, ( 28 ) designs generative personas drawing on the Big Five personality traits, proving that the LLM provides responses consistent with the assigned psychological profile. In ( 3 ) LLMs’ predictions are influenced by socio-demographic information, emphasizing that complex profiles have a larger influence than individual attributes in isolation. This is further supported in ( 59 ) , where the use of finer-grained personas regularly affects relevant textual surface properties, such as lexical consistency and dialogic fidelity. Computational Social Science. Computational social science aims to uncover the laws of emergent social behavior using computational and simulation-based methods ( 10 ) . Social simulations ( 52 ) have been implemented according to different paradigms, notably epidemiological models ( 37 ; 44 ) and ABM ( 40 ) . Recently, ABM has gained particular momentum for its bottom-up nature, where collective behavior emerges from the local interaction of software components conceptualized as agents . ABM simulations have been adopted for studying several social phenomena, including information diffusion ( 42 ) , emotion contagion ( 17 ) , epidemics ( 36 ) , economics ( 15 ) , human mobility ( 11 ) . LLM-based Agents. Integrating LLMs into ABM simulations is gaining increasing attention ( 53 ) . Despite being in its infancy ( 8 ) , several efforts have been made to systemize the field of LLM-based agents ( 18 ; 57 ; 24 ; 61 ) . Simulators mostly leverage

Read the complete text