ABCD-LINK : Annotation Bootstrapping for Cross-Document Fine-Grained Links
Serwar Basch, Ilia Kuznetsov, Tom Hope, Iryna Gurevych · 2025
Read ABCD-LINK : Annotation Bootstrapping for Cross-Document Fine-Grained Links by Serwar Basch, Ilia Kuznetsov, Tom Hope, Iryna Gurevych (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01387.txt.
Introduction
Links between documents at the level of single sentences matter for tasks such as tracing how news frames a story or how a review answers a paper. Serwar Basch, Ilia Kuznetsov, Tom Hope, and Iryna Gurevych introduce ABCD-LINK, a framework to build such link data from scratch in any domain. Hand annotation is slow, so bootstrapping is used to grow data with machine help.
The framework first generates and validates semi-synthetic linked documents, then uses them to benchmark and shortlist linking methods. The best methods are finally applied to natural text pairs with humans in the loop. In peer review and news domains, joining retrieval models with language models won a seventy-three percent human approval rate for suggested links, more than double strong retrievers alone. All code, data, and protocols are released.
Catalog record
| CREATOR | Serwar Basch, Ilia Kuznetsov, Tom Hope, Iryna Gurevych |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-01387
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ABCD-LINK : Annotation Bootstrapping for Cross-Document Fine-Grained Links Serwar Basch, Ilia Kuznetsov, Tom Hope, Iryna Gurevych Abstract Understanding fine-grained relations between documents is crucial for many application domains. However, the study of automated assistance is limited by the lack of efficient methods to create training and evaluation datasets of cross-document links. To address this, we introduce a new domain-agnostic framework for selecting a best-performing approach and annotating cross-document links in a new domain from scratch. We first generate and validate semi-synthetic datasets of interconnected documents. This data is used to perform automatic evaluation, producing a shortlist of best-performing linking approaches. These approaches are then used in an extensive human evaluation study, yielding performance estimates on natural text pairs. We apply our framework in two distinct domains -- peer review and news -- and show that combining retrieval models with LLMs achieves 78% link approval from human raters, more than doubling the precision of strong retrievers alone. Our framework enables systematic study of cross-document understanding across application scenarios, and the resulting novel datasets lay foundation for numerous cross-document tasks like media framing and peer review. We make the code, data, and annotation protocols openly available. 1 1 1 https://github.com/UKPLab/arxiv2025-ABCD-Link Figure 1: Framework overview. 1 Introduction Documents rarely exist in isolation. In many practical scenarios, understanding one document requires reasoning over its relationships with others. Fact-checkers might trace a claim to the specific sentences across multiple articles that support or contradict it ( Thorne et al. 2018 ; Wadden et al. 2020 ; Chen et al. 2024b ) . Authors and meta-reviewers reading peer reviews need to connect reviewer comments to the paper to decide on the course of action ( Kuznetsov et al. 2022 ; D’Arcy et al. 2024 ) . News analysts might use paraphrased or ideologically reframed sentences across different outlets to gain insights into reporting bias and source alignment ( Giorgi et al. 2023 ) . Yet finding such fine-grained relations can be cognitively demanding, motivating the need for machine assistance. Following Kuznetsov et al. 2022 , we call fine-grained relations between documents links , and the general task of identifying them linking . While automatic linking holds great potential, the progress has been limited due to the manual effort required to annotate and evaluate links, and the diversity of link types across domains. Thus, labeled corpora of links are scarce, preventing the development and evaluation of automated linking approaches at scale. To move beyond the bottleneck of full manual annotation, we propose a general framework for bootstrapping sentence-level cross-document linking without the need for pre-existing labeled data (Figure 1 ). While links might be hard for humans to detect , linked documents are easy to generate with state of the art LLMs. Building upon this insight, we (1) propose a method to generate and validate semi-synthetic corpora of linked documents. We use this data to (2) automatically evaluate a wide range of approaches to arrive at a shortlist of best-performing linking approaches. Finally, we use the best-performing approaches to (3) assist humans in labeling natural, non-synthetic document pairs in a large-scale human-in-the-loop evaluation study. Our framework allows practitioners to efficiently select the best linking approach given a domain and link type, while producing manually labeled many-to-many cross-document linking datasets for downstream applications. We assess our framework by applying it in two distinct domains, peer reviews and news articles, each with its own task-specific link type and underlying data. In total, we contribute: • A novel framework for efficient human-in-the-loop evaluation and labeling of links; • A data generation approach for creating semi-synthetic datasets of linked documents in the domain of interest, incl. human validation; • A large-scale analysis of state-of-the-art linking approaches, incl. a novel human evaluation protocol; • Two new manually annotated non-synthetic corpora of cross-document links in peer review and news. We release the code, datasets, and annotation protocols to support future research on sentence-level cross-document linking and its applications. 2 Related Work Numerous NLP tasks study or rely on cross-document relations , including cross-document coreference ( Miyabe et al. 2008 ; Cybulska and Vossen 2014 ; Ravenscroft et al. 2021 ) , fact-checking ( Thorne et al. 2018 ; Wadden et al. 2020 ; Chen et al. 2024b ) , scholarly peer review analysis ( Kuznetsov et al. 2022 ; D’Arcy et al. 2024 ) , and citation/source detection ( Syed et al. 2023 ; Liang et al. 2024 ) . While prior work focuses on limited relation types and relies on manual labeling, we contribute a general, structured framework for bootstrapping linking from scratch in new domains and for new link types. We evaluate our framework in two domains: peer reviews and news, which target challenging and underrepresented forms of linking, such as subjective evaluation and ideological framing, as opposed to encyclopedic and factual relations found in Wikipedia ( Feith et al. 2024 ) . From a machine-assisted annotation perspective, prior work reduces labeling effort by pre-selecting candidate links with heuristics Ravenscroft et al. 2021 , or simple retrieval methods, e.g. cosine similarity with a fixed cutoff ( Kuznetsov et al. 2022 ) . Our framework generalizes from this idea: we automatically benchmark various zero-shot approaches, and validate the results through human evaluation to select the best approach for the task and domain at hand. In that, our work contributes to the study of synthetic data in NLP ( Ding et al. 2024 ; Liu et al. 2022 ; Zhao et al. 2025 ; Josifoski et al. 2023 ; Veselovsky et al. 2023 ) . Møller et al. 2024 and Kazemi et al. 2025 find that synthetic data can match or complement human-labeled corpora in low-resource settings. To the best of our knowledge, we are the first to use synthetic data to study cross-document links. The use of synthetic data carries risks ( Van Breugel et al. 2023 ; Lu et al. 2023 ) . To mitigate them, our framework only uses synthetic data to automatically evaluate different linking approaches, employs a human validation step, and supplements evaluations on synthetic data with focused human evaluation, contributing to best practices in the use of synthetic data in NLP. Human judgment is essential for evaluating NLP systems, yet it is difficult to elicit reliably. There is growing interest in structured human evaluation protocols , particularly for tasks involving model-assisted annotation or subjective decision-making. Prior work examined inter-annotator calibration over time ( Uma et al. 2021 ) , preference-based assessment of model outputs ( Clark et al. 2021 ) , and the influence of interface design on annotation quality ( Klie et al. 2018 ) . In the context of LLM evaluation, human-in-the-loop setups have proven essential for capturing fine-grained distinctions and improving label quality ( Zhou et al. 2023 ) . Here, we contribute a novel and efficient annotation protocol that combines candidate pre-selection with a ranking-based comparison for a reliable assessment of top-performing approaches. 3 Setup To trade off between task complexity and utility, we focus on sentence-level linking , cast as a sentence pair classification task. Given two documents, the source document A=\{a_{1},\ldots,a_{N}\} and the target document B=\{b_{1},\ldots,b_{M}\} , the goal is to predict for each cross-document sentence pair (a_{i},b_{j}) whether a relation