SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations

2025

Read SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07405.txt.

Introduction

Manon Berriche and colleagues built SPOT, a French dataset for spotting comments that halt or redirect heated online threads. Sociologists call such moments stopping points. They include irony, quiet doubt, or short challenges that counterspeech research often misses.

The corpus holds more than forty three thousand hand-labeled Facebook comments tied to posts flagged as false information, with article and thread context attached. The team turned the idea into a yes-or-no labeling task with clear guidelines. Fine-tuned encoder models beat prompted large models in their benchmarks. Moderation researchers gain a new way to study how ordinary comments defuse false claims.

Catalog record

YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07405

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations Abstract We introduce SPOT ( Stopping Points in Online Threads ), the first annotated corpus translating the sociological concept of stopping point into a reproducible NLP task. Stopping points are ordinary critical interventions that pause or redirect online discussions through a range of forms — irony, subtle doubt or fragmentary arguments— that frameworks like counterspeech or social correction often overlook. We operationalize this concept as a binary classification task and provide reliable annotation guidelines. The corpus contains 43,305 manually annotated French Facebook comments linked to URLs flagged as false information by social media users, enriched with contextual metadata (article, post, parent comment, page or group, and source). We benchmark fine-tuned encoder models ( CamemBERT ) and instruction-tuned LLMs under various prompting strategies. Results show that fine-tuned encoders outperform prompted LLMs in F 1 F_{1} score by more than 10 percentage points, confirming the importance of supervised learning for emerging non-English social media tasks. Incorporating contextual metadata further improves encoder models F 1 F_{1} scores from 0.75 0.75 to 0.78 0.78 . We release the anonymized dataset, along with the annotation guidelines and code in our code repository , to foster transparency and reproducible research. Keywords: annotation, context-aware NLP, critical interventions, dataset, Facebook, French, online conversations, online moderation, social media, stopping point, pragmatics SPOT: An Annotated French Corpus and Benchmark for Detecting Critical Interventions in Online Conversations Manon Berriche *,1 , Célia Nouri *,1,2 , Chloé Clavel 2,3 , Jean-Philippe Cointet 1 * Equal contributions, 1 Sciences Po, médialab, 2 INRIA, ALMAnaCH, 3 Télécom Paris {manon.berriche, celia.nouri}@sciencespo.fr Abstract content 1. Introduction Research on online discourse has largely focused on phenomena perceived as harmful — such as polarization, misinformation or hate speech — and on their automated detection and measurement ( 15 ; 54 ; 50 ; 52 ) . A smaller but growing literature has examined how users respond to these harms through counter-speech, social correction, or user-led moderation ( 11 ; 18 ; 4 ) . In parallel, the Natural Language Processing (NLP) community has produced annotated datasets and benchmarks addressing related phenomena—from stance and (dis)agreement to counter-speech and corrective replies—along with annotation schemes and models designed to capture such behaviors ( 32 ; 7 ) . Yet much of this work tends to focus on explicit, goal-directed interventions (evidence-based refutations or collective moderation efforts) while overlooking far more common everyday reactions that do not fully correct or sanction a message but nevertheless interrupt, reframe, or temporarily stall its circulation. Still, they matter as they reveal forms of criticism that standard taxonomies often neglect and provide empirical windows into how communities self-negotiate norms and interpretive frames within a thread. However, they pose challenges for NLP as these interventions are often subtle, ironic, or fragmentary. To address this gap, we introduce and operationalize the sociological notion of a stopping point ( 3 ) : ordinary critical interventions that mark hesitation, resistance, or create a pause or shift in an online conversation without necessarily resolving the factual status of the contested content. Examples range from skeptical prompts ( “Is this true?” ) and dismissive asides ( “You’re talking nonsense” ) to terse denunciations ( “Report” ) or corrective replies with links, and can also take ironic forms ( “When pigs fly” , “Yeah, and I’m the Queen of England” ). Crucially, stopping points are defined by their conversational function —momentarily halting or redirecting the flow of interaction—rather than by tone, polarity, or factual accuracy, making them particularly challenging for both annotation and automatic detection. Indeed, as illustrated by Figure 1(a) , lexical cues alone are not sufficient to spot stopping points. Although the first two comments ( “This is completely absurd!” , “This is ridiculous” ) appear to be critical in tone, they take the post at face value. They thus express agreement rather than challenge or redirection and should be annotated as non–stopping points. In contrast, the third comment ( “Attention everyone: did you know that it’s now possible to report false information on Facebook?” ) reframes the discussion as a moderation issue and invites collective action. As such, it qualifies as a stopping point. In Figure 1(b) , the third reply ( “There will also be women’s bust sizes and men’s penis lengths!!” ) — responding to the first comment in Figure 1(a) — functions differently from a straightforward rebuttal but still serves as a stopping point. Lacking explicit critical markers, it uses irony and hyperbole to recast the original claim as an absurd slippery-slope caricature. This performative reframing contrasts with goal-directed interventions like social correction or counterspeech, which typically provide evidence, sources, or reasoned arguments to correct misinformation or combat hate speech. Together, these examples underscore that reliably annotating and detecting stopping points requires contextual understanding of the broader discussion, rather than sole reliance on lexical cues from isolated comments. (a) Top-level post and comments. (b) Replies under the first top-level comment. Figure 1 : An example thread from the SPOT corpus showing (a) the post with top-level comments, and (b) the reply thread under the first comment. Colors indicate page/group name (orange), domain name (pink), article title (green), non-stopping points (blue), and stopping points (red). English translations for (a) 1 1 1 Page/Group name : Top Tips; Domain name : TopsFromTheWeb.org; Article title : Your religion will soon have to be stated on your ID card (January 2018); Comment 1 : But where are we headed? This is completely absurd! I thought we weren’t supposed to stigmatize people? But to me, here, it’s the exact opposite!! In a secular country, every citizen has the right to practice their religion without having to answer to the government!!; Comment 2 : This is ridiculous, terrorist acts are carried out by suicide bombers who have nothing to do with religion; Comment 3 : Attention everyone: did you know that it’s now possible to report false information on Facebook? Take advantage of it! and (b) 2 2 2 Reply 1 Fuckers; Reply 2 No, don’t believe what they’re saying, it’s completely FALSE… rest assured; Reply 3 There will also be women’s bust sizes and men’s penis lengths!!! are provided below. This paper makes four contributions. First, we translate the sociological notion of stopping point into a reproducible annotation task and provide detailed annotation guidelines. Second, we present SPOT, a corpus of 43,305 manually annotated French Facebook comments linked to URLs flagged as false information. The dataset is enriched with parent-post and comment context, shared URL and source information, page or group features. Third, we benchmark automated approaches for the stopping point detection task by comparing a fine-tuned CamemBERT model ( 38 ) , trained with different combinations of textual and contextual features, to instruction-tuned Large Language Models (LLMs) evaluated through various prompting strategies. Our results show that fine-tuned encoders substantially outperform prompted LLMs on the SPOT corpus, underscoring the effectiveness of supervised domain adaptation for social media text analysis in non-English settings. We also find that incorporating the publication context significantly improves performance. Fourth, we analyze the types of case that are incorrectly predicted by current methods, providing empirical guidance for future work on improving modeling strategies for

Read the complete text