How Human Feedback Shapes AI-generated Community Notes

Soham De, Isaac Slaughter, Jiawei Guo, Qiao-Yun Cheng · 2026

Read How Human Feedback Shapes AI-generated Community Notes by Soham De, Isaac Slaughter, Jiawei Guo, Qiao-Yun Cheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30905.txt.

Introduction

Soham De, Isaac Slaughter, Jiawei Guo and Qiao-Yun Cheng analyse how human feedback reshapes AI-drafted Community Notes, the crowd-sourced fact checks shown on major social platforms. In the newer Collaborative Notes system on X, a language model drafts a note and human contributors then suggest edits. The team studied the full corpus: 19,146 collaborative notes and 211,850 pieces of human feedback. They built a taxonomy of the suggestions and tracked which kinds got incorporated into later versions.

Factual corrections and added context were most likely to be adopted; subjective policy judgments rarely were. Feedback made notes more helpful overall, most strongly when a suggestion challenged the draft's main claim, especially from active contributors. Yet collaborative notes still reached shown status less often than notes written by humans alone or AI alone, with thin participation the key bottleneck.

Catalog record

CREATORSoham De, Isaac Slaughter, Jiawei Guo, Qiao-Yun Cheng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30905

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

How Human Feedback Shapes AI-generated Community Notes Soham De, Isaac Slaughter, Jiawei Guo, Qiao-Yun Cheng Abstract Community Notes, a bridging-based crowd-sourced fact-checking system, has emerged as a new mechanism for moderating misleading information on social media and has been adopted by major platforms including X, Facebook, Instagram, Threads, and TikTok. Since its introduction, there has been an open question about what role AI could play in scaling and optimizing the system. Recently, X extended its Community Notes system by introducing Collaborative Notes : notes initially drafted by an LLM and iteratively refined based on feedback from human contributors. In this work, we systematically analyze the complete corpus of 19,146 collaborative notes and 211,850 instances of human feedback. First, we develop a taxonomy of human suggestions for improving AI-generated note drafts and find that suggestions involving factual corrections and additional context are most likely to be incorporated, while subjective policy judgments rarely are. Second, we examine changes in helpfulness across versions of collaborative notes and find that human feedback leads to more helpful notes, with the greatest impact coming from suggestions that challenge the main claim in the previous draft, particularly when submitted by more active contributors. Finally, we find that although collaborative notes improve through human feedback, they reach helpful status and are shown on the platform at lower rates than human-only or AI-only notes, with limited human participation emerging as a key bottleneck. Nevertheless, rather than serving as a weaker substitute, collaborative notes tend to play a complementary role, predominantly targeting posts that do not attract human-only or AI-only notes. Our analysis provides an initial description of efforts to use AI to improve crowdsourced content moderation in a real-world moderation system and outlines pathways for future improvements to such features. 1 Introduction Community Notes has emerged as the industry-standard mechanism for providing additional context on potentially misleading social media content. In this system, fact-checking is not performed by professional fact-checkers or platforms. Instead, ordinary users of the platform step up to request additional context on potentially misleading posts, propose fact-checking notes and also rate on proposed fact-checking notes, representing a bottom-up approach to content moderation. Originally launched as Birdwatch ( 30 ) on X (then, Twitter) in 2019, the approach has since been adopted in various forms by most major platforms, including YouTube, Facebook, Instagram, Threads and TikTok. Studies investigating Community Notes find that the notes are effective at reducing the engagement with and spread of misinformation when attached to posts ( 6 ; 27 ) , but also highlight persistent challenges around coverage ( 29 ) and the speed at which notes are attached to posts ( 27 ) . Several directions for overcoming these challenges have been proposed in the literature, ranging from algorithmic improvements to the scoring process ( 12 ) to the integration of AI in various parts of the system ( 18 ) . These AI-assisted approaches include enhancing existing human-written notes ( 7 ; 32 ) as well as developing more autonomous, end-to-end systems for automated note generation ( 31 ; 17 ) . To retain the system’s credibility and trust, any attempts to scale Community Notes with AI need to be designed in ways that preserve human judgment in determining what the community ultimately finds helpful. Since the inception of Community Notes, X has made several attempts to incorporate AI into the note-writing workflow. In June 2025, the platform released an API allowing bots to author draft notes on posts flagged as potentially misleading by users. This allows Community Notes contributors to design their own AI-powered fact-checking bots that can automatically generate note drafts, which are then rated by humans before being shown publicly. More recently, in February 2026, X introduced Collaborative Notes , an extension of the system in which an LLM (Grok) automatically drafts a community note on flagged content, with the draft iteratively refined through helpfulness ratings and free-text suggestions submitted by Community Notes contributors. Collaborative human-AI note-writing offers several advantages over human-only note-writing. LLM assistance may produce more thoroughly researched drafts than individual contributors could create on their own, while also incorporating human-provided context that may be critical in fast-evolving situations. It also simplifies the task from writing a note from scratch to suggesting edits, which may lower the barrier to participation and broaden engagement across the contributor community. At the same time, the collaborative process raises new questions: (i) whether the back-and-forth between human suggestions and LLM revisions actually improves note quality, (ii) whether it does so effectively, and (iii) whether the resulting notes are ultimately more helpful than those written through existing workflows. In this work, we present a systematic analysis of the human-AI collaborative notes on X, aiming to characterize the current state of the system and inform future design decisions. We center our analysis on the following research questions: RQ1 What feedback do humans provide on AI-generated notes? Specifically, who are the raters that engage in this system and what kinds of suggestions do they provide? What kinds of suggestions are more likely to be adopted by the LLM as it revises new versions? RQ2 How do collaborative notes evolve over time? Specifically, do collaborative notes become more or less helpful as they are rewritten based on the human feedback? What drives improvement between versions of collaborative notes? RQ3 Are collaborative notes effective? Specifically, how do collaborative notes perform compared to non-collaborative human- or AI-written notes on the platform? What are potential explanations for their relative performance? To answer these questions, we collected the complete corpus of all collaborative notes on X up to May 5, 2026 resulting in 19,146 note versions on 10,600 unique posts, and 211,850 ratings and suggestions, over a span of 3 months. We performed a mixed-methods analysis, using qualitative methods to develop a taxonomy to characterize the suggestions provided by humans and quantitative methods to describe the evolution of collaborative notes over time, and how they compare against non-collaborative notes. In doing so, we make the following contributions: 1. A qualitative analysis resulting in a taxonomy of the kinds of suggestions provided by human raters on AI-drafted notes, and the prevalence of each type, including which are most likely to be incorporated into the next version (Section 5.1 ). We find that participation is dominated by experienced contributors, and that suggestions grounded in factual corrections, added context, or source critique are the most likely to be incorporated, while subjective policy judgments are rarely incorporated. 2. An analysis characterizing how a collaborative note improves over time , and how that depends on the overall chain length, kinds of suggestions provided, and the experience and political lean of the raters providing them (Section 5.2 ). We find that while notes generally improve with sustained revision, improvement is rarely monotonic, and that longer revision chains are associated with significantly greater quality gains. 3. A comparative analysis examining the effectiveness of collaborative notes against non-collaborative human- and AI-written notes , and highlighting potential explanations and opportunities for improvement (Section 5.3 ). We find that collaborative notes currently underperform both baselines on helpfulness and take longer to reach rater consensus, with a lack of ratings emerging as a key structural bottleneck and point to

Read the complete text