Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 CCS: Human-centered comp

Shuning Zhang, Linzhi Wang, Dai Shi, Yuwei Chuai, Jingruo Chen, Yunyi Chen, Yifan Wang, Yating Wang, Xin Yi, Hewu Li · 2025

Read Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 CCS: Human-centered comp by Shuning Zhang, Linzhi Wang, Dai Shi, Yuwei Chuai, Jingruo Chen, Yunyi Chen, Yifan Wang, Yating Wang, Xin Yi, Hewu Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-11052.txt.

Introduction

Ordinary commenters fact-check faster than official systems. Shuning Zhang and eight co-authors prove it at scale in this 2025 study. Community Notes on X attaches crowd-written corrections to misleading posts. Notes arrive slowly, though. The authors analyzed over two million comments. They covered eighteen hundred misleading posts that eventually received notes. A language model pipeline was trained to spot fact-checking comments. Those are replies that correct a post with reasons or evidence.

Nearly every misleading post drew its first fact-checking comment early. Over ninety-nine percent beat the official note's creation. The median first correction took about six minutes. The median note took nearly ten hours. Posts that eventually displayed notes carried significantly more such comments, too. The authors argue for a change. Platforms should treat early corrective comments as an early-warning signal. They are also a resource for faster, synthesized fact checks.

Catalog record

CREATORShuning Zhang, Linzhi Wang, Dai Shi, Yuwei Chuai, Jingruo Chen, Yunyi Chen, Yifan Wang, Yating Wang, Xin Yi, Hewu Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-11052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY Price: 15.00 ISBN: 978-1-4503-XXXX-X/18/06 CCS: Human-centered computing Social media CCS: Information systems Crowdsourcing CCS: Human-centered computing Empirical studies in collaborative and social computing CCS: Security and privacy Human and societal aspects of security and privacy Shuning Zhang, Linzhi Wang, Dai Shi, Yuwei Chuai, Jingruo Chen, Yunyi Chen, Yifan Wang, Yating Wang, Xin Yi, Hewu Li Abstract . Community-based fact-checking is promising to reduce the spread of misleading posts at scale. However, its effectiveness can be undermined by the delays in fact-check delivery. Notably, user-initiated organic comments often contain debunking information and have the potential to help mitigate this limitation. Here, we investigate the feasibility of synthesizing comments to generate timely high-quality fact-checks. To this end, we analyze over 2.2 million replies on X and introduce Commenotes, a two-phase framework that filters and synthesizes comments to facilitate fact-check delivery. Our framework reveals that fact-checking comments appear early and sufficiently: 99.3% of misleading posts receive debunking comments within the initial two hours since post publication, with synthesized commenotes successfully earning user trust for 85.8% of those posts. Additionally, a user study (N=144) found that the synthesized commenotes were often preferred, with the best-performing model achieving a 70.1% win rate over human notes and being rated as significantly more helpful. Figure 1 . The algorithm flow of Commenotes, which could be decomposed into two stages: filtering and synthesizing. During filtering, we prompted fine-tuned Language Models (LMs) to identify fact-checking comments. During synthesizing, we used LLMs to generate commenotes for these posts. 1. Introduction The spread of online misinformation remains a critical challenge, with far-reaching and oftentimes negative effects on societies ( Xu et al. 2023 ; Yu et al. 2024 ) . Social media platforms are under pressure to innovate in their content moderation strategies for effectively fact-check misleading information at scale. Among emerging strategies, community-based fact-checking has gained prominence ( Pröllochs 2022 ; Drolsbach et al. 2024 ; Chuai et al. 2024a ) . For instance, \mathbb{X} has rolled out Community Notes system to enable users collaboratively add context annotations to potentially misleading posts ( Pröllochs 2022 ) . Community-based fact-checking is valued for its effectiveness, transparency, and ability to incorporate diverse perspectives, thereby fostering a degree of trust ( Drolsbach et al. 2024 ; Gao et al. 2024 ) . Despite these strengths, a key limitation of community-based fact-checking, such as current implementation of Community Notes on \mathbb{X} , is the speed to generate fact-checks at the early stage of misinformation diffusion ( Chuai et al. 2024a ) . In-depth investigation of claims can be time-consuming, often taking many hours or even days ( Guo et al. 2022 ) . This challenge is compounded by the limited number of available fact-checkers, making it infeasible to scrutinize every piece of content. Consequently, fact-checkers are often forced to prioritize overtly false or intentionally misleading claims over material that is more intricate or nuanced ( Robertson et al. 2020 ) . These leads to a significant time lag between post creation and note creation, with many community notes coming after the half-life of posts’ spreading ( Chuai et al. 2024a ; Pfeffer et al. 2023 ) . This delay allows misinformation to spread widely before corrective notes appear, severely limiting the effectiveness of the intervention ( Chuai et al. 2024b ) . Recent advances in Large Language Models (LLMs) show promise for synthesizing user-initiated debunking materials into timely fact-checking notes ( De et al. 2025 ) . We posit that the vast volume of user comments, often generated moments after a post appears, contains valuable, early-emerging debunking information. To systematically assess the feasibility of harnessing this information with LLMs, we first analyze the informational landscape of user comments, then introduce our framework, and finally evaluate its effectiveness. This leads to our first research question (RQ1): RQ1. Do comments on misleading posts contain debunking information, and are these debunking comments created earlier than the display of community notes? If comments to misleading posts do contain factual corrections or fact-checking content, we proceed to our second research question (RQ2): RQ2. Can large language models (LLMs) effectively summarize the fact-checking information in comments and generate community-note–style fact-checks that are helpful to users? Based on a large-scale dataset of N=2,225,260 comments across 1,841 source posts from \mathbb{X} ’s Community Notes platform ( Chuai et al. 2025 ) , we analyzed whether the comments contain debunking information, and their validity in generating high-quality fact-checks. For RQ1, we found that 73.1% of posts received fact-checking comments within the first hour, and a median of 36.5 such comments appeared before the corresponding community note was displayed. We further found a strong positive correlation between post popularity and the number of fact-checking comments, indicating the potential of comment-based fact-checking for viral posts. Motivated by the high volume and rapid emergence of these user-generated fact-checks, we developed Commenotes, a two-stage framework designed to harness these comments. The framework first employs a filtering stage, using fine-tuned LMs to automatically filter relevant fact-checking comments of a post. Subsequently, a synthesis stage prompts LLMs to consolidate the information from the filtered comments into a concise, synthesized commenote 1 1 1 Note that we used the upper-cased Commenotes to denote the two-stage framework, and lower-cased commenotes to denote the synthesized notes, similar to Community Notes vs. community notes. . The framework was carefully designed to ensure accurate information extraction, minimize bias, and improve efficiency in note generation. Towards RQ2, we conducted extensive technical evaluations and a user study. Because the choice of LLM in the synthesis stage may substantially influence output quality and biases, we treated model choice as a key factor, comparing four distinct models: gpt-4o, claude-3.7-sonnet, grok-4, and qwen2.5-72b-instruct. We found that using the comments within just two hours, 85.8% of the commenotes are already useful, and 73.9% of the commenotes are useful using the first 60 comments of a post for synthesizing. A subsequent robustness evaluation revealed that, on average, over 90% of all commenotes were rated as either “helpful” or “somewhat helpful” across all experimental conditions. Notably, commenotes showed high helpfulness even for posts without existing community notes: 92.0% of notes created for posts lacking a displayed note and 91.2% for posts with no notes at all were rated as at least “somewhat helpful.” A detailed analysis of note characteristics further confirmed a strong overall performance, with the average score combining all models exceeding 3.70 out of 5 on most quality dimensions. These results demonstrated the feasibility of Commenotes to timely and reliably synthesize comments into commenotes. In the subsequent user study (N=144), we compared commenotes against human-written community notes ( De et al. 2025 ) , and found that most models produced notes of comparable or superior quality. Notably, grok-4 achieved a 70.1% win rate over human notes and was significantly preferred in terms of clarity, coverage, and impariality. Furthermore, the effectiveness of the notes such as helpfulness showed no systematic correlation with post topics or user demographics like political stance, indicating the robustness of our approach across varied contexts. In summary, this

Read the complete text