Can LLMs Automate Fact-Checking Article Writing?

Dhruv Sahnan David Corney Irene Larraz Giovanni Zagni, Ruben Miguez Zhuohan Xie Iryna Gurevych Elizabeth Churchill, Tanmoy Chakraborty Preslav Nakov · 2025

Read Can LLMs Automate Fact-Checking Article Writing? by Dhruv Sahnan David Corney Irene Larraz Giovanni Zagni, Ruben Miguez Zhuohan Xie Iryna Gurevych Elizabeth Churchill, Tanmoy Chakraborty Preslav Nakov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-17684.txt.

Introduction

Fact-checkers write articles that explain a verdict to the public, but automatic systems usually output only a label. A team of authors asked in 2025 whether language models can write those articles. They first interviewed experts at leading fact-checking groups to learn what a good article needs, then built a system, QRAFT, that mimics the human writing workflow step by step.

Professional fact-checkers judged the output in a human evaluation. QRAFT beat several earlier text-generation methods, yet fell well short of articles written by experts. The authors present the work as opening a new research direction rather than solving it, and they release their code for others to build on.

Catalog record

CREATORDhruv Sahnan David Corney Irene Larraz Giovanni Zagni, Ruben Miguez Zhuohan Xie Iryna Gurevych Elizabeth Churchill, Tanmoy Chakraborty Preslav Nakov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-17684

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs Automate Fact-Checking Article Writing? Dhruv Sahnan David Corney Irene Larraz Giovanni Zagni, Ruben Miguez Zhuohan Xie Iryna Gurevych Elizabeth Churchill, Tanmoy Chakraborty Preslav Nakov Abstract Automatic fact-checking aims to support professional fact-checkers by offering tools that can help speed up manual fact-checking. Yet, existing frameworks fail to address the key step of producing output suitable for broader dissemination to the general public: while human fact-checkers communicate their findings through fact-checking articles, automated systems typically produce little or no justification for their assessments. Here, we aim to bridge this gap. In particular, we argue for the need to extend the typical automatic fact-checking pipeline with automatic generation of full fact-checking articles . We first identify key desiderata for such articles through a series of interviews with experts from leading fact-checking organizations. We then develop Qraft , an LLM-based agentic framework that mimics the writing workflow of human fact-checkers. Finally, we assess the practical usefulness of Qraft through human evaluations with professional fact-checkers. Our evaluation shows that while Qraft outperforms several previously proposed text-generation approaches, it lags considerably behind expert-written articles. We hope that our work will enable further research in this new and important direction. The code for our implementation is available at https://github.com/mbzuai-nlp/qraft.git . 1 Introduction Figure 1: Our proposed pipeline for automatic fact-checking. We extend the typical steps in the automatic claim fact-checking pipeline to include a new task: fact-checking article generation . According to a March 2024 survey Pew Research Center (2024) , nearly 80% of American adults on major social media platforms regularly encounter news-related content. While these platforms allow the rapid spread of breaking news , they have also been heavily misused to circulate dubious claims. In response, the role of fact-checkers has grown to be increasingly vital. However, manual fact-checking efforts cannot match the scale of global dis/misinformation, prompting a push towards automating the process Vlachos and Riedel (2014) . As shown in Figure 1 , most existing studies frame automatic fact-checking to comprise five main steps: ( i ) check-worthy claim detection, ( ii ) verified claim retrieval, ( iii ) evidence retrieval, ( iv ) claim verification, and ( v ) brief explanation generation Vlachos and Riedel (2014) ; Nakov et al. (2021a) ; Nakov et al. (2021b) ; Guo et al. (2022) . Note that some studies skip ( i ), ( ii ) and ( v ). In practice, fact-checkers do more than just verifying claims; they also publish detailed fact-checking articles that guide readers toward a clear understanding of a claim by presenting factual arguments and explaining how they lead to the final verdict Graves (2017) . Yet existing automatic fact-checking pipelines fail to account for this key time-consuming step of the manual fact-checking workflow. One may argue that brief explanation generation serves as the automated counterpart to fact-checking article writing; however, expert requirements for high-quality articles transcend the traditional definition of fact-checking explanations Warren et al. (2025) . Such explanations are typically concise and aim to justify the factuality decision Kotonya and Toni (2020) . In contrast, fact-checking articles usually aim not just to provide the evidence leading to the verdict, but also to give the readers the necessary background and contextual information, to explain the origins of the claim, to clarify all likely interpretations of the claim, to address counter-arguments, etc. We thus argue that the typical automatic fact-checking pipeline needs to be extended with an extra step, automatic generation of a fact-checking article , as shown in Figure 1 . Moreover, it has repeatedly been highlighted that automatic fact-checking overlooks insights from professional fact-checkers Nakov et al. (2021a) ; Juneja and Mitra (2022) ; Das et al. (2023) . Thus, we study the task of generating fact-checking articles in close collaboration with experts from several world-leading fact-checking organizations, aiming to understand the key elements of a good fact-checking article. We further provide a brief analysis of why professional fact-checkers generally do not trust Artificial Intelligence (AI) to fully automate this task. Finally, based on what we have learned from this study, we develop Qraft , a framework using multiple AI agents that collaborate to write and iteratively refine a fact-checking article, designed as the first attempt at a comprehensive solution to cater to the requirements highlighted by the experts. Our framework leverages large language models (LLMs) as its foundation, as they can generate fluent, long-form text ( Yang et al., 2023 ; Xie et al., 2023 ; Wang et al., 2025 ) . Recent studies have highlighted that agentic workflows can further enhance performance, particularly for tasks involving the generation of documents grounded in a set of facts ( Huot et al., 2025 ; Shao et al., 2024 ; Wang et al., 2024a ) . Thus, we design Qraft as a multi-agent collaboration that mimics the fact-checking article writing process of human experts. Qraft breaks the writing process down into two main stages. In the first stage, Qraft gathers evidence nuggets relevant to the claim, formulates an outline, and then populates it to produce an initial draft. In the second stage, Qraft simulates an editorial review that uses conversational question-answering interactions between LLM agents to formulate a list of edits to refine the draft and to ensure professional standards of writing. We further benchmark Qraft against several state-of-the-art text generation frameworks using automatic evaluation strategies. In addition, we conduct evaluations with professional fact-checkers focusing on assessing Qraft ’s real-world usefulness. We find that Qraft shows better performance on all automatic metrics; yet, the human evaluations reveal key limitations, indicating the need for constant expert supervision when using LLM-based frameworks for this task. To sum up, our contributions are as follows: • We introduce the novel task of automatically generating full fact-checking articles, working in close collaboration with experts from leading fact-checking organizations to enrich the task with their insights. • We propose Qraft , a unique framework designed to generate a fact-checking article given ( i ) a claim, ( ii ) its veracity, and ( iii ) a set of evidence documents; we further show that Qraft outperforms several generic text generation approaches. • We conduct expert evaluations to assess the practical usefulness of Qraft and find that it falls short of expert-written articles. We also discuss the key limitations as highlighted by experts to facilitate future research. 2 What Expert Fact-Checkers Want from a Fact-Checking Article “ A fact-checking article for a claim is a form of communication from a fact-checker to the public that provides the necessary context around the claim, argues its veracity, and explains why it may or may not be exactly as claimed. ” – An experienced fact-checker It is crucial for fact-checkers to maintain public trust in the fact-checking process, which means that they need to be extremely careful when reporting data towards fact-checking a claim. Characteristic Comments Accurately clarifies the claim P3 \xrightarrow{} “ We must have clarity regarding the veracity of the claim and understand the reasons behind it after reading the article. ” P2 \xrightarrow{} “ The article must accurately convey why the evidence contradicts the claim, providing all the necessary context. ” Origin of the claim P4 \xrightarrow{} “ The article must present the

Read the complete text