The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the

Xingyu Deng, Xi Wang, Mark Stevenson · 2025

Read The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the by Xingyu Deng, Xi Wang, Mark Stevenson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20844.txt.

Introduction

Xingyu Deng, Xi Wang, and Mark Stevenson present a 2025 perspective paper arguing that scientific fact-checking needs a new kind of evidence search. Checking a scientific claim means digging through full papers, not just abstracts, and the evidence hides in long documents full of tables, figures, and jargon, on topics that shift as science moves. Existing systems sidestep all that with small abstract datasets.

The authors map five challenges. Retrieval must follow the claim's semantics rather than keywords, and time-aware search must track citations so stale findings get retired. Parsers should exploit document structure, complex expressions like tables need handling, and a source's credibility must be judged. Early experiments back each challenge. Their call: build retrieval systems designed for real papers from the start.

Catalog record

CREATORXingyu Deng, Xi Wang, Mark Stevenson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-20844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) (ICTIR ’25), July 18, 2025, Padua, Italy DOI: 10.1145/3731120.3744614 ISBN: 979-8-4007-1861-8/2025/07 CCS: Information systems Specialized information retrieval Xingyu Deng, Xi Wang, Mark Stevenson Abstract . Scientific fact-checking aims to determine the veracity of scientific claims by retrieving and analysing evidence from research literature. The problem is inherently more complex than general fact-checking since it must accommodate the evolving nature of scientific knowledge, the structural complexity of academic literature and the challenges posed by long-form, multimodal scientific expression. However, existing approaches focus on simplified versions of the problem based on small-scale datasets consisting of abstracts rather than full papers, thereby avoiding the distinct challenges associated with processing complete documents. This paper examines the limitations of current scientific fact-checking systems and reveals the many potential features and resources that could be exploited to advance their performance. It identifies key research challenges within evidence retrieval, including (1) evidence-driven retrieval that addresses semantic limitations and topic imbalance (2) time-aware evidence retrieval with citation tracking to mitigate outdated information, (3) structured document parsing to leverage long-range context, (4) handling complex scientific expressions, including tables, figures, and domain-specific terminology and (5) assessing the credibility of scientific literature. Preliminary experiments were conducted to substantiate these challenges and identify potential solutions. This perspective paper aims to advance scientific fact-checking with a specialised IR system tailored for real-world applications. † † cc-license: by 1. Introduction Fact-checking aims to assess the veracity of factual claims based on credible evidence ( Guo et al. 2022 ; Zeng et al. 2021 ) and serves as a crucial safeguard for mitigating misinformation. Scientific fact-checking is a specialised variant of this task, grounded in scientific knowledge, with the objective of combating misinformation that affects the public, helping researchers in knowledge discovery and assisting individuals in understanding scientific advancements ( Vladika and Matthes 2023 ) . This is particularly important given the rapid emergence of new scientific findings, where both professionals and the public must assess the credibility of information. A prominent case occurred during the COVID-19 pandemic, in which politically motivated misinformation—ranging from inflated infection statistics to unsupported treatments—circulated extensively, eroding public trust and endangering health communication ( Luengo and García-Marín 2020 ) . However, existing approaches to scientific fact-checking remain limited, primarily relying on the retrieval of evidence from relatively simple and small-scale sources ( Wadden et al. 2020 ; Mohr et al. 2022 ; Wang et al. 2023b ; Bulian et al. 2020 ; Saakyan et al. 2021 ; Sarrouti et al. 2021 ; Kotonya and Toni 2020 ; Wadden et al. 2022a ) . For example, SciFact-Open ( Wadden et al. 2022a ) , the largest available dataset for scientific fact-checking, contains 500,000 documents – substantially smaller than PubMed, which contains over 37 million biomedical publications. In addition, SciFact-Open consists only of abstracts, rather than full-text papers, thereby excluding critical structural and citation information, ignoring long-range context and scientific expression conveyed through tables and figures. These design simplifications may hinder the applicability of current approaches in real-world settings, where scientific evidence is embedded in long and structurally complex documents with multimodal content. Fact-checking is a knowledge-intensive task, where the verification process relies on sourcing evidence from a reliable upstream Information Retrieval (IR) system. Emerging findings indicate the value of effective retrieval in improving fact-checking systems. For example, introducing even a small amount of noise into evidence can significantly degrade fact-checking performance ( Sauchuk et al. 2022 ) . Recent Retrieval-Augmented-Generation (RAG) techniques have been widely used for fact-checking ( Dmonte et al. 2024 ; Raina and Gales 2024 ; Momii et al. 2024 ; Sevgili et al. 2024 ; Singal et al. 2024 ; Ullrich et al. 2024 ; Khaliq et al. 2024 ) , where retrieval models are fine-tuned to identify high-quality evidence for claim verification. These observations underscore the critical role of robust evidence retrieval, as an ideal IR system for fact-checking should rank all relevant evidence at the top while filtering out non-evidential noise. Ensuring retrieval robustness is crucial to maintaining sufficient yet relevant evidence, which is essential for improving scientific fact-checking accuracy. A major trend in fact-checking research is to consider realistic settings that employ rich, diverse and timely evidence sources, as seen in FEVER (using Wikipedia) ( Thorne et al. 2018a ) and AVeriTeC (using web-wide resources) ( Schlichtkrull et al. 2024b ) . In document level evidence retrieval , current general fact-checking systems over-rely on commercial search APIs, which do not consider the specific requirements of fact-checking ( Zeng et al. 2021 ; Thorne et al. 2018b ; Schlichtkrull et al. 2024a ) . Such reliance on commercial search APIs – with limited adaptability – has left document retrieval methodologies under-explored in fact-checking, especially for domain-specific corpora such as scientific fact-checking. Current scientific fact-checking systems primarily employ off-the-shelf IR methods ( Vladika and Matthes 2023 ) , such as lexical matching and semantic relevance ranking, which do not scale effectively for large-scale scientific corpora. In addition, the distribution of relevant evidence across scientific topics is highly imbalanced, which degrades both retrieval effectiveness and efficiency, especially for claims with scarce supporting literature. SciFact-Open ( Wadden et al. 2022a ) , which extends the original SciFact dataset ( Wadden et al. 2020 ) for large-scale evaluation, illustrates this issue: verification performance on SciFact-Open drops by at least 15 F1 points for all well-performed fact-checking systems developed in SciFact ( Wadden et al. 2022a ) . While increasing corpus size enhances evidence diversity, it also amplifies retrieval noise, reducing efficiency in both retrieval and verification. Beyond that, high semantic relevance does not guarantee high evidential relevance, and irrelevant yet semantically similar documents can introduce noise into downstream verification ( Zhang et al. 2023 ) . These challenges underscore the necessity of developing tailored document retrieval systems specifically designed for scientific fact-checking, as effective retrieval is a prerequisite for accurate claim verification. Beyond document-level evidence retrieval, within-document evidence retrieval is also essential for processing complex scientific literature. Scientific papers, unlike general fact-checking documents, are long, structured, domain-specific and involve additional metadata. As scientific fact-checking evolves from abstract-based to full-paper retrieval, retrieval models must account for metadata (e.g., publish date, citations) and complex structured data format (e.g., charts, tables, figures). This necessitates the adaptation of verification models such as SciBERT ( Beltagy et al. 2019 ) for domain-specific terminologies ( Wadden et al. 2020 ) , Longformer ( Beltagy et al. 2020 ) for long-range dependencies ( Wadden et al. 2022b ) and TAPAS ( Herzig et al. 2020 ) for tabular data verification ( Akhtar et al. 2022 ) . Scientific expressions in academic papers are highly structured and contextually interdependent, where textual content, tabular data, and figures mutually reinforce the conveyed information. However, existing scientific fact-checking systems primarily operate at the abstract level, adopting methodologies similar to general fact-checking ( Vladika and Matthes 2023 ; Wadden et al. 2020 ; Li et al. 2021

Read the complete text