COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations

Rui Xing Preslav Nakov Timothy Baldwin Jey Han Lau · 2025

Read COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations by Rui Xing Preslav Nakov Timothy Baldwin Jey Han Lau (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24810.txt.

Introduction

Rui Xing, Preslav Nakov, Timothy Baldwin, and Jey Han Lau study crowd notes that flag posts as false on big social sites. Such notes help when they are clear, but most notes are never shown; the queue is slow and what makes a note help is vague. Their set, COMMUNITYNOTES, is built to probe that help.

The set holds more than one hundred thousand posts with user notes and help marks, in more than one tongue. Their frame also grows and tunes plain word rules for why a note helps, with an auto prompt loop. Those tuned rules lifted scores on both help calls and the why calls. The work gives fact check teams a firmer base.

Catalog record

CREATORRui Xing Preslav Nakov Timothy Baldwin Jey Han Lau
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24810

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations Rui Xing Preslav Nakov Timothy Baldwin Jey Han Lau Abstract Fact-checking on major platforms, such as X, Meta, and TikTok, is shifting from expert-driven verification to a community-based setup, where users contribute explanatory notes to clarify why a post might be misleading. An important challenge here is determining whether an explanation is helpful for understanding real-world claims and the reasons why, which remains largely underexplored in prior research. In practice, most community notes remain unpublished due to slow community annotation, and the reasons for helpfulness lack clear definitions. To bridge these gaps, we introduce the task of predicting both the helpfulness of explanatory notes and the reason for this. We present COMMUNITYNOTES , a large-scale multilingual dataset of 104k posts with user-provided notes and helpfulness labels. We further propose a framework that automatically generates and improves reason definitions via automatic prompt optimization, and integrate them into prediction. Our experiments show that the optimized definitions can improve both helpfulness and reason prediction. Finally, we show that the helpfulness information are beneficial for existing fact-checking systems. The code and data are available at: https://github.com/ruixing76/Helpfulness-FCExp . Figure 1: An example of X Community Notes. The user-generated note appears as “ Readers added context .” 1 Introduction Beyond simple verification, fact-checking systems should provide explanations that clarify why a claim is misleading ( Warren et al. 2025 ) . Explanations not only strengthen user trust, but also mitigate pitfalls of fact-checking, e.g., backfire, where the false belief is reinforced rather than hindered ( Lewandowsky et al. 2012 ; Guo et al. 2022 ; Xing et al. 2025 ) . Typically, fact-checking explanations are produced by expert organizations. Yet the ecosystem is currently shifting, and platforms are increasingly experimenting with community-based fact-checking, where contributors collaboratively provide explanatory notes on claims ( Augenstein et al. 2025 ) . This crowdsourcing process distributes the labor of verification and offers diverse perspectives from the community. A representative example is Community Notes on X (formerly Twitter), which allows users to attach notes explaining potentially misleading posts ( Figure 1 ). 1 1 1 https://communitynotes.x.com/guide/ Critically, for a note to become visible or formally published, it must receive sufficient amount of ratings from a diverse set of users ( De et al. 2025 ) . A key challenge in that respect is to decide whether an explanation is helpful for understanding real-world claims and why, which is underexplored in previous research on fact-checking explanations ( Atanasova et al. 2020 ; Guo et al. 2022 ; Russo et al. 2023 ; Eldifrawi et al. 2024 ) . This process is also slow on Community Notes : notes can take from hours to days to appear, and the vast majority (over 90%) never become visible, despite the invested contributors efforts. Moreover, while the platform provides reasons for judging notes as helpful or unhelpful, these reasons lack clear definitions, leaving both raters and contributors uncertain about the evaluation criteria ( De et al. 2025 ; Augenstein et al. 2025 ) . This opacity hinders the efficiency, scalability, and explainability of crowd-sourced fact-checking. Motivated by the above, here we explore the idea of simplifying and making more explicit the helpfulness annotation process. Our contributions are as follows: • We introduce the task of predicting the helpfulness and its reasons given a note and the original X post. • We present COMMUNITYNOTES , a large-scale explanation helpfulness dataset containing over 104K potentially misleading posts and corresponding user-provided notes. • We propose a framework that automatically generates and optimizes reason label definitions via prompt optimization; integrating them to helpfulness/reason predictors to improve their performance. • We apply helpfulness prediction to the evidence sufficiency task ( Atanasova et al. 2022 ) and further incorporate it as auxiliary signals into automated fact-checking on CLIMATE-FEVER ( Diggelmann et al. 2020 ) , showing its potential to enhance existing fact-checking systems. 2 Related Work Community-Based Fact-Checking The proliferation of misinformation has become a growing societal concern, posing significant risks to areas such as public health, science, and politics ( Borenstein et al. 2025 ) . To address this challenge, fact-checking has emerged as a central strategy in mitigating the spread of false or misleading information ( Guo et al. 2022 ) . Traditionally, fact-checking has been conducted by professional third-party organizations that evaluate the veracity of claims through expert analysis. However, recent years have witnessed a paradigm shift toward more decentralized, community-based approaches ( Renault et al. 2024 ; Augenstein et al. 2025 ) . Major social media platforms have increasingly embraced this model. X’s Community Notes has become the most prominent example, enabling users to collaboratively annotate and assess the accuracy of online content. Following this trend, Meta recently terminated its partnerships with professional fact-checkers in favor of a similar community-driven system ( Catalanello and Sanders 2025 ) , while TikTok has also introduced a comparable feature ( TikTok 2025 ) . Empirical evidence suggests that community-based fact-checking can be highly effective in countering misinformation. Studies have shown that the presence of a Community Note can reduce the rate of retweets by nearly half ( Renault et al. 2024 ) , while displaying such notes on misleading posts has been found to decrease their spread by an average of 61% ( Chuai et al. 2024 ) . Moreover, authors of posts flagged by community notes are up to 80% more likely to delete their content, and such posts generally exhibit markedly lower virality ( Drolsbach and Pröllochs 2023 ; Renault et al. 2024 ) . However, Community Notes often requires a long time to gather sufficient contributor ratings to determine helpfulness, and the absence of reason definitions further undermines the trustworthiness of the mechanism. Label Definitions Recent work has shown that providing models with explicit definitions of labels can lead to substantial improvements in classification. Khatuya et al. 2025 used predefined label descriptions and trained the model to generate these descriptions based on the input text, achieving sizable improvements in multi-label classification. Similarly, Gao et al. 2023 demonstrated that curating a small dictionary of terms for label description can improve zero-shot accuracy by 17-19% absolute, while also increasing the robustness to prompts. Peskine et al. 2023 showed improvements when using label definitions with GPT-3 zero-shot classification on the challenging task of fine-grained conspiracy theory detection. Automatic Prompt Optimization Prompt engineering has become essential to many Natural Language Processing (NLP) tasks; yet, manual trial-and-error approaches remain a barrier to effective use. Automatic Prompt Optimization (APO) methods address this challenge by systematically searching for high-performing prompts ( Ramnath et al. 2025 ) . These approaches explore the large instruction space using diverse strategies: PromptBreeder evolves task-specific prompts across generations using optimized mutation strategies ( Fernando et al. 2024 ) ; PromptAgent applies Monte Carlo Tree Search (MCTS) with feedback reflection to guide exploration ( Wang et al. 2024 ) ; and SCULPT represents prompts as trees, enabling targeted refinements while preserving overall structure—especially useful for long or complex prompts ( Kumar et al. 2025 ) . 3 Predicting Whether an Explanation is Helpful and Why 3.1 Task Definition Given a post P that contains potentially misleading information and a note N

Read the complete text