Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs
Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, Vera Schmitt · 2025
Read Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs by Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, Vera Schmitt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-08909.txt.
Introduction
Premtim Sahitaj and co-authors tested large language models as automated fact-checkers in a 2025 paper. Human fact-checking is careful but slow, while false claims spread fast. The team asked how well off-the-shelf models judge real-world claims under different labeling schemes.
They ran three sizes of the Llama-3 model family on nearly 18,000 PolitiFact claims spanning 2007 to 2024. Evidence came from restricted web searches, and the models also had to explain their verdicts. Larger models classified better and justified better without any fine-tuning. Small models given one example rivaled fine-tuned ones. The study sets baselines for a task where explanation quality matters as much as the label.
Catalog record
| CREATOR | Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, Vera Schmitt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-08909
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Automated Fact-Checking of Real-World Claims: Exploring Task Formulation and Assessment with LLMs Premtim Sahitaj, Iffat Maab, Junichi Yamagishi, Jawan Kolanowski, Sebastian Möller, Vera Schmitt Abstract Fact-checking is necessary to address the increasing volume of misinformation. Traditional fact-checking relies on manual analysis to verify claims, but it is slow and resource-intensive. This study establishes baseline comparisons for Automated Fact-Checking (AFC) using Large Language Models (LLMs) across multiple labeling schemes (binary, three-class, five-class) and extends traditional claim verification by incorporating analysis, verdict classification, and explanation in a structured setup to provide comprehensive justifications for real-world claims. We evaluate Llama-3 models of varying sizes (3B, 8B, 70B) on 17,856 claims collected from PolitiFact (2007–2024) using evidence retrieved via restricted web searches. We utilize TIGERScore as a reference-free evaluation metric to score the justifications. Our results show that larger LLMs consistently outperform smaller LLMs in classification accuracy and justification quality without fine-tuning. We find that smaller LLMs in a one-shot scenario provide comparable task performance to fine-tuned Small Language Models (SLMs) with large context sizes, while larger LLMs consistently surpass them. Evidence integration improves performance across all models, with larger LLMs benefiting most. Distinguishing between nuanced labels remains challenging, emphasizing the need for further exploration of labeling schemes and alignment with evidences. Our findings demonstrate the potential of retrieval-augmented AFC with LLMs. [orcid=0000-0003-3908-5681, [email protected], ] *1 [[email protected], ] [[email protected], ] [[email protected], ] [orcid=0000-0003-3057-0760, [email protected], ] [orcid=0000-0002-9735-6956, [email protected], ] keywords Automated Fact-Checking ,Large Language Models ,Retrieval-Augmented Generation , † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: ROMCIR 2025: The 5th Workshop on Reducing Online Misinformation through Credible Information Retrieval (held as part of ECIR 2025: the 47th European Conference on Information Retrieval), April 10, 2025, Lucca, Italy † † address: Technische Universität Berlin, Quality and Usability Lab, Berlin, 10587, Germany † † address: Deutsches Forschungszentrum für Künstliche Intelligenz, Speech and Language Technology Lab, Berlin, 10559, Germany † † address: National Institute of Informatics, Digital Content and Media Sciences Research Division, Tokyo, 101-8430, Japan † † address: Harz University of Applied Sciences, Faculty of Automation and Computer Science, Wernigerode, 38855, Germany † † corresp: Corresponding author. 1 Introduction Misinformation, whether spread inadvertently or with the intention to deceive, is a global challenge and can only be mitigated effectively through fact-checking efforts Lewandowsky et al. 2020 . Generally, fact-checking is defined as the assessment of the truthfulness of a check-worthy claim Vlachos and Riedel 2014 ; Hassan et al. 2017 . For fact-checking to be effective, fact-checking itself must be convincing and justified Guo et al. 2022 . A well-known source of human-verified knowledge is PolitiFact 1 1 1 https://politifact.com , where experts manually identify check-worthy claims from news and social media and document their verification efforts in written articles. Traditional fact-checking of these claims relies on human-driven exploration, analysis, and conclusion. Consequently, this process is rather slow and expensive, lagging behind the rapid spread of misinformation. Delayed fact-checking efforts allow false narratives to take hold, distort reality, and influence public opinion, a vulnerability that is often exploited by bad actors Nyhan 2020 . AFC systems assist human efforts to combat misinformation by leveraging state-of-the-art techniques from areas such as Natural Language Processing (NLP), Natural Language Generation (NLG), and Information Retrieval (IR). Ideally, these systems automatically extract claims from the presented media, retrieve relevant and credible references, and provide evidence-based verdicts on the aggregated results. As opposed to style-based detection approaches that learn to distinguish claims based on writing patterns, AFC systems follow a knowledge-based approach that relies on verification knowledge to make judgements on claims Zhou and Zafarani 2021 . Expert fact checkers can utilize AFC systems as intelligent decision support assistance to eliminate repetitive manual tasks, highlight inconsistencies, and present their findings Graves 2018 . Humans often distrust fact-checking work that challenges their beliefs, perceiving them as biased or manipulated Brandtzaeg and Følstad 2017 . This skepticism is likely to be aggravated with closed systems, where the lack of transparency around internal mechanisms and design decisions further erodes trust. Brandtzaeg and Følstad 2017 argue that to strengthen trust, fact-checking processes must be made transparent. LLMs such as GPT-4, Claude 3.5 Sonnet, and Llama-3, have provided significant potential for a broad range of text-to-text reasoning tasks. Integrating LLMs as an inference engine into AFC systems may enhance transparency by generating veracity predictions and the accompanying natural language explanations. However, Setty 2024 demonstrate that, for AFC-related classification tasks, fine-tuned small language models (SLM) outperform LLMs. This indicates that further research is needed to effectively utilize LLMs for AFC. This paper investigates the task formulation and assessment for AFC of real-world claims with LLMs to establish baselines in various settings and to evaluate whether truthfulness ratings can be effectively modeled or if alternative approaches to claim annotations and task formulation are necessary. Based on these findings, future approaches can make more informed design choices and improve the reliability and effectiveness of AFC systems. We propose a framework for AFC with LLMs in a few-shot setup without model fine-tuning for claim analysis, claim veracity prediction, and the generation of justifications as natural language explanations. In the scope of this work, we assess the performance of our framework on 17,856 real-world claims from PolitiFact based on three labeling schemes, with or without web evidence, and across models of different sizes (i.e. 3B, 8B, 70B). Using reference-free evaluation metrics and conducting extensive experiments, we provide insight into how evidence integration, model size, and labeling complexity impact system performance. Additionally, we consider fine-tuning small state-of-the-art classification models for estimating the upper bound of predictive performance extractable from different components of the data points in the collected dataset and assess the performance of LLMs with a few-shots relative to this limit. Thus, our findings contribute to the development of more robust and transparent AFC systems using LLMs. 2 Related Work NLP provides the foundation for efficiently processing and interpreting text, making it critical for addressing misinformation detection. Advances in transformer-based architectures have significantly improved language modeling and generation. Vaswani et al. Vaswani et al. 2023 introduced the transformer architecture, which leverages multi-head self-attention to capture contextual dependencies between tokens. Variants such as BERT Devlin et al. 2019 and GPT Radford et al. 2018 illustrate encoder-only and decoder-only applications, respectively. BERT is well-suited for sequence classification tasks, while GPT generates text autoregressively for sequence-to-sequence tasks. In the context of this study, we refer to these lightweight architectures as SLMs Nguyen et al. 2024 . Kaplan et al. 2020 showed that the loss of a language scales as a power law with model parameter size, pre-training dataset size, and compute budget, guiding the development of current large language models (LLMs). Most LLMs follow the GPT architecture with modifications, aligning their b ,ehavior to specific tasks through instruction fine-tuning, reinforcement learning from human feedback (RLHF) Ouyang et al. 2022 , direct preference optimization (DPO) Rafailov et al. 2023 , or supervised fine-tuning. Prompts serve as inputs to these models, acting as task instructions. Effective prompt engineering, including template design and task-specific examples in a few-shot scenario, enhances their utility Schulhoff et al. 2024 ;