KG20C & KG20C-QA: Scholarly Knowledge Graph Benchmarks for Link Prediction and Question Answering
Hung-Nghiep Tran, Atsuhiro Takasu · 2025
Read KG20C & KG20C-QA: Scholarly Knowledge Graph Benchmarks for Link Prediction and Question Answering by Hung-Nghiep Tran, Atsuhiro Takasu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21799.txt.
Introduction
Hung-Nghiep Tran and Atsuhiro Takasu release two benchmark datasets for question answering over scholarly data. KG20C is a knowledge graph built from the Microsoft Academic Graph, using targeted venue selection, quality filtering, and a defined schema. It had circulated informally before; this paper is its first formal, peer-reviewed description.
KG20C-QA sits on top of the graph. Templates convert graph triples, the subject-relation-object facts, into natural-language question and answer pairs. The benchmark works both for graph-based models, such as knowledge graph embeddings, and for text-based models, such as large language models. The authors benchmark standard embedding methods on it, break performance down by relation type, and supply reproducible evaluation protocols. Their stated aim is a reusable, extensible resource for research on question answering, reasoning, and knowledge-driven applications in the scholarly domain, where clean benchmarks have been scarce.
Catalog record
| CREATOR | Hung-Nghiep Tran, Atsuhiro Takasu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-21799
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KG20C & KG20C-QA: Scholarly Knowledge Graph Benchmarks for Link Prediction and Question Answering Hung-Nghiep Tran, Atsuhiro Takasu Abstract In this paper, we present KG20C and KG20C-QA, two curated datasets for advancing question answering (QA) research on scholarly data. KG20C is a high-quality scholarly knowledge graph constructed from the Microsoft Academic Graph through targeted selection of venues, quality-based filtering, and schema definition. Although KG20C has been available online in non-peer-reviewed sources such as GitHub repository, this paper provides the first formal, peer-reviewed description of the dataset, including clear documentation of its construction and specifications. KG20C-QA is built upon KG20C to support QA tasks on scholarly data. We define a set of QA templates that convert graph triples into natural language question–answer pairs, producing a benchmark that can be used both with graph-based models such as knowledge graph embeddings and with text-based models such as large language models. We benchmark standard knowledge graph embedding methods on KG20C-QA, analyze performance across relation types, and provide reproducible evaluation protocols. By officially releasing these datasets with thorough documentation, we aim to contribute a reusable, extensible resource for the research community, enabling future work in QA, reasoning, and knowledge-driven applications in the scholarly domain. The full datasets will be released at https://github.com/tranhungnghiep/KG20C/ upon paper publication. I Introduction Scholarly knowledge graphs (KGs) provide structured representations of the publication ecosystem, capturing entities such as authors, papers, venues, affiliations, and citations. Unlike scientific data, which encodes domain-specific facts (e.g., chemical formulas or biomedical interactions), scholarly data encodes the metadata of the scientific process itself: who wrote which paper, in which venue, under which affiliation, and citing which works. Such information underlies applications including bibliographic search, expert finding, trend analysis, and recommendation systems [ 1 , 2 ] . With the increasing availability of large-scale scholarly graphs such as the Microsoft Academic Graph (MAG) [ 2 ] , it becomes important to establish principled benchmark datasets that enable reproducible evaluation of algorithms for reasoning over scholarly metadata. Benchmark datasets have been central to progress in knowledge graph representation learning. Widely used resources such as WN18RR [ 3 , 4 ] and FB15k-237 [ 5 , 6 ] have defined standard evaluation settings for link prediction and completion. However, these benchmarks are lexical or encyclopedic in nature and do not reflect the structure or challenges of scholarly metadata and its applications. Meanwhile, large scholarly graphs such as MAG are massive, noisy, and lack standardized train/validation/test splits. Without curated datasets and controlled evaluation settings, it is difficult to ensure fair comparison and reproducibility. We present KG20C , a curated benchmark scholarly knowledge graph. KG20C is constructed in three stages: (1) data extraction from MAG, focusing on well-defined scholarly entity and relation types; (2) graph construction with careful cleaning and pruning to reduce noise; and (3) dataset splitting into disjoint training, validation, and test sets, following established best practices to avoid data leakage issues observed in earlier benchmarks such as WN18 and FB15k [ 7 ] . The result is a high-quality benchmark dataset, directly comparable in scope and usage to WN18RR and FB15k-237 [ 3 , 5 ] , but grounded in the scholarly domain. While preliminary versions of KG20C were informally available, this paper provides its first formal peer-reviewed introduction, ensuring a stable, citable, and reproducible resource. Building on KG20C, we introduce KG20C-QA , a question answering (QA) benchmark over scholarly data. QA has emerged recently as a central task in the era of large language models, providing a natural means of evaluating reasoning ability over structured and textual resources [ 8 , 9 ] . In the current stage, KG20C-QA consists of one-hop questions grounded in the KG20C schema. It is released in both natural language and entity–relation forms, enabling evaluation of both text-based methods (e.g., LLMs) and graph-based methods (e.g., knowledge graph embeddings). The current release focuses on one-hop QA, defined using single KG triple templates. While one-hop QA may appear simple, this work provides the foundation of QA benchmark grounded in controlled, high-quality scholarly data and prepare for multi-hop QA and reasoning benchmarks in the future. Our contributions are as follows: • We construct and officially release KG20C , a curated scholarly knowledge graph derived from MAG, with high-quality entity and relation coverage, careful cleaning, and standardized train/validation/test splits. • We construct and release KG20C-QA , a benchmark dataset for question answering on scholarly data, provided in both natural language and entity–relation forms. • We establish reproducible baseline evaluations for link prediction and QA tasks on these datasets, demonstrating their utility as challenging and standardized benchmarks. By releasing KG20C and KG20C-QA, our goal is to provide standardized resources that enable the community to advance methods for reasoning on scholarly data. We expect these benchmarks to support a broad spectrum of research, from classical knowledge graph embeddings to modern QA with large language models. II Related Work II-A Knowledge Graph Benchmarks Benchmark datasets have been central to advancing representation learning on knowledge graphs. Early resources such as WN18 and FB15k provided lexical and encyclopedic graphs, but were later shown to suffer from test leakage through inverse relations, leading to inflated scores [ 7 ] . Their corrected variants, WN18RR and FB15k-237, introduced filtered splits to remove leakage and remain widely used as standard testbeds for link prediction [ 3 , 5 ] . Other encyclopedic graphs such as YAGO [ 10 ] and DBpedia [ 11 ] provide rich coverage but lack standardized splits suitable for controlled embedding evaluation. II-B Scholarly Knowledge Graphs Several large-scale scholarly graphs have been developed to capture the publication ecosystem, including the Microsoft Academic Graph (MAG) [ 2 , 1 ] and AMiner [ 12 ] . These resources span millions of entities and relations across authors, papers, venues, and affiliations, and are widely used for bibliometrics and scientometrics. However, they are massive, noisy, and not released as curated benchmarks with standardized training, validation, and test splits. Without such controlled datasets, reproducibility and fair comparison across methods remain limited. II-C QA Benchmarks on Knowledge Graphs Question answering (QA) over knowledge graphs has become an important evaluation paradigm, especially for natural language interfaces. Established QA benchmarks include SimpleQuestions [ 13 ] , WebQuestions [ 14 ] , MetaQA [ 15 ] , and LC-QuAD [ 16 ] , which provide natural language queries grounded in encyclopedic KGs such as Freebase or DBpedia. These datasets support both single-hop and multi-hop reasoning and have shaped much of the recent progress in KG-based QA. Recently, QA datasets have also been developed for scholarly data. SciQA [ 17 ] introduces questions grounded in the Open Research Knowledge Graph (ORKG), covering scholarly entities and scientific concepts, though it is relatively small in scale and focused on ORKG’s comparison tables. DBLP-QuAD [ 18 ] provides one-hop and multi-hop QA benchmarks derived from the DBLP scholarly graph, constructed using query templates and paraphrasing. While highly relevant, DBLP-QuAD is tailored to DBLP and does not align with the rigorous standards of KG completion benchmarks such as WN18RR or FB15k-237. II-D Our Contribution in Context Our work provides another curated benchmark resource. KG20C introduces a high-quality scholarly knowledge graph derived from MAG, constructed with careful cleaning and standardized splits that follow best practices to