Relation Extraction Across Entire Books to Reconstruct Community Networks: The AffilKG Datasets

Erica Cai, Sean McQuade, Kevin Young, Brendan O’Connor · 2025

Read Relation Extraction Across Entire Books to Reconstruct Community Networks: The AffilKG Datasets by Erica Cai, Sean McQuade, Kevin Young, Brendan O’Connor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10798.txt.

Introduction

Erica Cai, Sean McQuade, Kevin Young, and Brendan O'Connor build data sets to test knowledge graphs pulled from whole books in a 2025 paper. Graphs built by machine from text are used for later analysis, but judging their accuracy needs data where the true graph is large and connected. Past sets were too small, too broken up, or too complex.

Their AffilKG collection pairs full book scans with large labeled graphs in six data sets. The graphs record who belongs to which group, a form useful for studies of migration and community life. Three sets add wider relation types. Early tests show model scores vary a lot by set, which lets users trace how errors spread.

Catalog record

CREATORErica Cai, Sean McQuade, Kevin Young, Brendan O’Connor
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10798

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Relation Extraction Across Entire Books to Reconstruct Community Networks: The AffilKG Datasets Erica Cai, Sean McQuade, Kevin Young, Brendan O’Connor Abstract When knowledge graphs (KGs) are automatically extracted from text, are they accurate enough for downstream analysis? Unfortunately, current annotated datasets can not be used to evaluate this question, since their KGs are highly disconnected, too small, or overly complex. To address this gap, we introduce AffilKG 1 1 1 https://doi.org/10.5281/zenodo.15427977 , which is a collection of six datasets that are the first to pair complete book scans with large, labeled knowledge graphs. Each dataset features affiliation graphs, which are simple KGs that capture Member relationships between Person and Organization entities—useful in studies of migration, community interactions, and other social phenomena. In addition, three datasets include expanded KGs with a wider variety of relation types. Our preliminary experiments demonstrate significant variability in model performance across datasets, underscoring AffilKG ’s ability to enable two critical advances: (1) benchmarking how extraction errors propagate to graph-level analyses (e.g., community structure), and (2) validating KG extraction methods for real-world social science research. Figure 1: Pipeline of converting scans of historical text to affiliation graphs, where highlighted text indicates clubs. 1 Introduction Extracting social networks from text is of great interest to social scientists for understanding how elite communities interact O’Brien 2025 ; Broad 1996 and how network structures change across time Wellman and Wetherell 1996 ; Kossinets and Watts 2006 . Historically, many networks have been manually curated with high precision Edwards et al. 2020 , such as financial acquisitions from news articles Gugler et al. 2003 ; Clougherty et al. 2014 or interpersonal relationships in literary sagas Mac Carron and Kenna 2013 . However, manual annotation is labor-intensive and difficult to scale, while automated approaches often lack rigorous evaluation Drury et al. 2022 , raising concerns about their reliability for downstream analyses. In parallel, hundreds of papers on relation extraction (RE), a.k.a. knowledge graph (KG) extraction, aim to automate the construction of these networks as KGs, which represent information using triplets (e.g., \langle Bob ; member ; IEEE \rangle ), each of which encodes a relation ( member ) between two entities ( Bob , IEEE ) that is expressed in text. In a KG, entities are represented as nodes and relations as edges (Fig. 1 ) Ding et al. 2021 ; Nadgeri et al. 2021 ; Xu and Barbosa 2019 ; Trisedya et al. 2019 ; Han et al. 2021 . Despite prior poor large language model (LLM) performance for RE Jimenez Gutierrez et al. 2022 , recent works show increasingly better performance with limited training data Wadhwa et al. 2023 ; Wan et al. 2023 ; Zhou et al. 2024 ; Zhang and Soh 2024 ; Tao et al. 2024 ; Xue et al. 2024 ; Zhang et al. 2024 ; Rajpoot and Parikh 2023 . However, a fundamental question remains: Do widely used evaluations for RE and KG extraction Zhao et al. 2024 ; Detroja et al. 2023 reliably indicate performance on higher-level, practical graph analyses, such as identifying key individuals in a social network Cai et al. 2024 ? Most existing NLP datasets for RE make it difficult to assess how KG extraction errors affect downstream graph analyses. Many datasets label triplets that fail to define connected KGs because the underlying text consists of unrelated sentences and disjoint entities Zhang et al. 2017 ; Han et al. 2018 ; Riedel et al. 2010 ; Chen et al. 2021 ; Gardent et al. 2017 ; Huguet Cabot et al. 2023 . Others involve highly complex KGs with hundreds of relation types or label KGs in short documents, resulting in small, disconnected graphs Yao et al. 2019 . Even when datasets yield connected KGs, low extraction performance (often below 50 F1 points) limits meaningful insights into downstream analyses that are relevant for real world applications Luan et al. 2018 ; Jain et al. 2020 . To address these challenges, we introduce AffilKG , the first collection of datasets that pair full scanned books—along with their text extracted via optical character recognition (OCR)—with large, labeled knowledge graphs. Each of the six datasets in AffilKG provides an affiliation graph, which is a bipartite KG where edges denote Member relationships between Person and Club entities. These graphs are particularly valuable for studying community cohesion, interaction dynamics Prem Sankar et al. 2015 ; Tisch and Ischinsky 2023 , and broader topics such as social stratification and network evolution Corradi et al. 2024 ; Bichler et al. 2015 ; Young and Ready 2015 ; Grund and Densley 2015 . Three datasets are further expanded to include triplets of 4 to 11 additional relation types, such as College and Yacht Name , enabling richer relational analyses. The texts in AffilKG are unique archival repositories of information on elite families in specific locations. Known as ‘social registers’, these texts document elite individuals from diverse geographic regions and time periods, including the Southern US Whitmarsh 1951 , Denver 1931 , Rhodesia 1963 , Boston 1903 , Miami Jackson 1965 , and New York City 1892 . Social register texts served to connect elite families with one another in a common directory of an area, typically listing family members, addresses, club memberships and educational backgrounds. As a historical text, a given social register thus provides a comprehensive record of a local social network, making it ideal for evaluating RE methods their downstream impact on graph analyses. Affiliation graph triplets were generated either through manual annotation or by applying regular expressions and rule-based algorithms, which were rigorously validated against manually constructed ground truth. AffilKG addresses critical limitations in existing datasets and provides a foundation for studying how RE errors impact downstream graph analyses. We present preliminary results using both widely used NLP evaluation metrics and higher-level analyses relevant to real-world applications. Our findings indicate that, while extraction performance can be high enough to yield meaningful insights, there is also substantial variation across settings. This suggests that further investigation into the propagation of RE errors within graph analyses could uncover important patterns and provide actionable knowledge for practical use. 2 AffilKG Dataset Construction AffilKG contains six datasets pairing scanned historical social register books with large, labeled knowledge graphs. Each individual dataset includes an affiliation graph, and three provide expanded KGs with additional relation types. All datasets include high-quality scans, OCR text, and graph annotations, enabling robust evaluation of relation extraction and higher-level graph analyses (Fig. 1 ). Loc./Year # pages # nodes (person / org) # edges NYC/1892 299 5553 / 178 14061 Boston/1903 150 2534 / 188 6228 Denver/1931 34 358 / 248 1352 S. US./1951 All (13 states) 826 6188 / 3107 13262 Median: Tenn. 47 399 / 174 872 Rhod./1963 169 898 / 767 2367 Miami/1965 503 3775 / 1471 11830 Table 1: Details about Denver , Southern US (13 states aggregated), Tennessee (state with the median # of person nodes), Rhodesia , Miami , Boston , and New York City text/affiliation graph pairs. Loc./Year # nodes (person / other) # edges # rel types NYC/1892 13054 / 5182 29263 4 Boston/1903 6748 / 4619 18945 5 Miami/1965 6340 / 8065 30845 11 Table 2: Details about Miami ,

Read the complete text