Scalable Evaluation of Online Facilitation Strategies via Synthetic Simulation of Discussions

Dimitris Tsirmpas, Ion Androutsopoulos, John Pavlopoulos · 2025

Read Scalable Evaluation of Online Facilitation Strategies via Synthetic Simulation of Discussions by Dimitris Tsirmpas, Ion Androutsopoulos, John Pavlopoulos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16505.txt.

Introduction

Dimitris Tsirmpas, Ion Androutsopoulos, and John Pavlopoulos design systems that make synthetic online discussions for research. Tests with human participants are costly, so simulated ones could allow cheap pilot tests. The authors give a theoretical, task agnostic framework for the design, testing, and running of such systems.

Their case study is online facilitation, where a facilitator steers discussion rather than just moderating content. Tests show small quantized open models can simulate well at a fraction of the cost of large proprietary models. Synthetic pilots reveal a key flaw in language model facilitators before human trials begin. They intervene too often, since they cannot tell when to step in. That flaw derails the talks in patterns like those seen with humans.

Catalog record

CREATORDimitris Tsirmpas, Ion Androutsopoulos, John Pavlopoulos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16505

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Scalable Evaluation of Online Facilitation Strategies via Synthetic Simulation of Discussions Dimitris Tsirmpas, Ion Androutsopoulos, John Pavlopoulos Abstract Limited large-scale evaluations exist for facilitation strategies of online discussions due to significant costs associated with human involvement. An effective solution is synthetic discussion simulations using LLM to create initial pilot experiments. We propose a simple, generalizable, LLM -driven methodology to prototype the development of LLM facilitators, and produce high-quality synthetic data without human involvement. We use our methodology to test whether current facilitation strategies can improve the performance of LLM facilitators. We find that, while LLM facilitators significantly improve synthetic discussions, there is no evidence that the application of more elaborate facilitation strategies proposed in modern Social Science research lead to further improvements in discussion quality, compared to more basic approaches. Additionally, we find that small LLM (such as Mistral Nemo 12B) can perform comparably to larger models (such as LLaMa 70B), and that special instructions must be used for instruction-tuned models to induce toxicity in synthetic discussions. We confirm that each component of our methodology contributes substantially to high quality data via an ablation study. We release an open-source framework SynDisco 1 1 1 https://github.com/dimits-ts/synthetic_discussion_framework (pip install syndisco) , which implements our methodology. We also release the “ VMD ( VMD )” a large, publicly available dataset containing LLM -generated and LLM -annotated discussions using multiple open-source LLM . 2 2 2 https://paperswithcode.com/dataset/vmd 1 Introduction Research on conversational moderation/facilitation techniques is crucial for adapting to ever-changing and demanding online environments. Relevant work traditionally focused on isolating and removing toxic and inappropriate content Seering 2020 ; Cresci et al. 2022 , whereas the current social media environment demands moderation systems to adequately explain their actions and prevent problematic user behavior before it surfaces Cho et al. 2024 ; Seering 2020 ; Cresci et al. 2022 ; Amaury and Stefano 2022 . Facilitation mechanisms are also needed to support community deliberation and group decision-making Kim et al. 2021 ; Seering 2020 . Note that “content moderation” usually involves flagging and removing content, as opposed to “conversational moderation”, which is studied in this paper. The terms “facilitation” and “conversational moderation” are otherwise equivalent Argyle et al. 2023 ; Korre et al. 2025 ; Falk et al. 2021 and we treat them as synonyms in this paper. Figure 1: LLM user-agents with distinct SDB participate in a discussion, while the LLM moderator monitors and attempts to improve the quality of the discussion. We need to design prompts and configurations for both types of LLM agents. A major challenge in connecting facilitation research to real-world needs is the substantial costs required both in researching and moderating discussions, due to human participation Rossi et al. 2024 . Many social media platforms overcome this by outsourcing moderation to volunteers or their own users Matias 2019 ; Schaffner et al. 2024 , while others support only conventional content moderation using traditional ML ( ML ) models, which are not enough in practice Horta Ribeiro et al. 2023 ; Schaffner et al. 2024 . LLM ( LLM ) have been hypothesized to be capable of facilitation tasks, which often require actively participating in the discussions, instead of passively flagging or removing content Small et al. 2023 ; Korre et al. 2025 . While studies exist for simulating user interactions in social media Park et al. 2022 ; Mou et al. 2024 ; Törnberg et al. 2023 ; Rossetti et al. 2024 ; Balog et al. 2024 , and for using LLM facilitators Kim et al. 2021 ; Cho et al. 2024 , none so far have combined the two approaches. We posit that synthetic simulations can be a cheap and fast way to develop and test preliminary experiments with LLM facilitators, initial versions of which may be unstable or unpredictable Atil et al. 2025 ; Rossi et al. 2024 , before testing them with human participants. Our work thus asks the following two questions: (1) Can we produce high-quality synthetic discussions, involving alternative facilitation strategies, by crafting an appropriate environment for simulations? (2) Can we boost the effectiveness of LLM facilitators (in synthetic discussions) using prompts aligned with facilitation strategies proposed in modern Social Science research? We propose a simple and generalizable methodology (§ 3 ) using LLM -driven synthetic experiments for online facilitation research, enabling fast and inexpensive model “debugging” and parameter testing (e.g., finding LLM facilitator instructions) without human involvement (Fig. 1 ). An ablation study (§ 5.2 ) demonstrates that each component of our methodology substantially contributes to generating high-quality data. We examine (§ 4 ) four LLM facilitation strategies based on current Social Science facilitation research, including a novel strategy with additional inspiration from RL ( RL ), and compare them with two common facilitation setups (no facilitation, LLM with simplistic prompts). We find that: (1) the presence of LLM facilitators has a positive and statistically significant influence on the quality of synthetic discussions, (2) facilitation strategies inspired by Social Science research often do not manage to outperform simpler strategies (§ 5.1 ). Furthermore, we release SynDisco, an open-source Python framework for generating and evaluating synthetic discussions, alongside the “ VMD ’’ 3 3 3 https://paperswithcode.com/dataset/vmd a large, publicly available dataset comprising automatically evaluated synthetic discussions (§ 6 ). We use open-source LLM and include all relevant configurations in order to make our study as reproducible as possible (see § A.3 , § A.5 ). 2 Background and Related Work 2.1 LLMs as Human Subjects When conducting social experiments with LLM instead of human subjects, it is imperative to know how representative results can be. Grossmann et al. 2023 argue that synthetic agents have the potential to eventually replace human participants, a perspective shared by other researchers Törnberg et al. 2023 ; Argyle et al. 2023 . Indeed, LLM have demonstrated complex, emergent social behaviors Park et al. 2023 ; Marzo et al. 2023 ; Leng and Yuan 2024 ; Abdelnabi et al. 2024 ; Abramski et al. 2023 , and are able to infer survey responses from SDB Hewitt et al. 2024 and personalized interviews Park et al. 2024 . However, significant limitations of LLM remain in the context of Social Science experiments. Issues include undetectable behavioral hallucinations Rossi et al. 2024 ; sociodemographic, statistical and political biases Anthis et al. 2025 ; Hewitt et al. 2024 ; Rossi et al. 2024 , often amplified during discussions Taubenfeld et al. 2024 ; unreliable survey responses Jansen et al. 2023 ; Bisbee et al. 2024 ; Neumann et al. 2025 ; inconsistent annotations Gligori’c et al. 2024 ; non-deterministic outputs Atil et al. 2025 , especially in closed-source models Bisbee et al. 2024 ; and excessive agreeableness due to alignment procedures Park et al. 2023 ; Anthis et al. 2025 ; Rossi et al. 2024 . Despite these issues, researchers frequently anthropomorphize LLM agents Rossi et al. 2024 , obscuring the true causes of their behavior Anthis et al. 2025 ; Zhou et al. 2024a . Our study must thus be conservative towards the generalizability of our results to discussions with humans. We stress that our methodology is designed for “debugging” and exploring LLM facilitators in-silico, before testing them in much more costly experiments with

Read the complete text