A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networks
Othmane Kabal \affilnum 1, Mounira Harzallah \affilnum 1, Fabrice Guillet \affilnum 1, Hideaki Takeda \affilnum 2, Ryutaro Ichise \affilnum 3 · 2026
Read A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networks by Othmane Kabal \affilnum 1, Mounira Harzallah \affilnum 1, Fabrice Guillet \affilnum 1, Hideaki Takeda \affilnum 2, Ryutaro Ichise \affilnum 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05476.txt.
Introduction
When a knowledge graph is built automatically from text, errors creep in everywhere. Othmane Kabal and four co-authors introduce a benchmark that judges graph building and graph learning together. Without that pairing, a poor result could come from the graph, the learning model, or both, and nobody could tell which.
Their benchmark uses one biomedical text collection and three graphs: two built by different automatic methods, and one curated by experts as a quality ceiling. Models are tested on node classification under a shared protocol. The setup gives future extraction methods and learning models a fair, reproducible yardstick.
Catalog record
| CREATOR | Othmane Kabal \affilnum 1, Mounira Harzallah \affilnum 1, Fabrice Guillet \affilnum 1, Hideaki Takeda \affilnum 2, Ryutaro Ichise \affilnum 3 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networks Othmane Kabal \affilnum 1, Mounira Harzallah \affilnum 1, Fabrice Guillet \affilnum 1, Hideaki Takeda \affilnum 2, Ryutaro Ichise \affilnum 3 Abstract Knowledge graphs automatically constructed from text are increasingly used in real-world applications. However, their inherent noise, fragmentation, and semantic inconsistencies significantly affect the performance of Graph Neural Networks (GNNs) on downstream tasks. Assessing their performance and robustness remains difficult, as it is often unclear whether observed results stem from the learning model or from the quality of the constructed graph itself. In this work, we introduce a dual-purpose benchmark designed to jointly evaluate (i) the performance of GNNs on noisy, text-derived graphs and (ii) the effectiveness of graph construction methods on a downstream task. The benchmark is built in the biomedical domain from a single textual corpus and includes two automatically constructed graphs generated using different extraction methods, alongside a high-quality reference graph curated by experts that serves as an upper performance bound. This design enables controlled comparison of construction methods and systematic evaluation of GNN robustness through semi-supervised node classification. We further provide a standardized, reproducible, and extensible evaluation framework, facilitating the integration of new graph extraction methods and learning models. keywords Knowledge Graphs, Graph Neural Networks, Benchmarking, Text-Derived Knowledge Graphs, Graph Representation Learning, Knowledge Graph Evaluation † † runninghead: Kabal et al. † † affiliation: \affilnum 1Nantes University, LS2N, Nantes, 44300, France \affilnum 2National Institute of Informatics, Chiyoda-ku, Tokyo, 101-8430, Japan \affilnum 3Institute of Science Tokyo, Tokyo, 152-8550, Japan † † corresponding: Othmane Kabal, Nantes University, LS2N, 44300 Nantes, France. 1 Introduction Knowledge graphs (KGs) play an increasingly central role in a wide range of artificial intelligence applications [ 12 , 21 ] . When combined with Graph Neural Networks (GNNs), they have demonstrated strong performance across numerous tasks, including node classification [ 29 ] , relation prediction [ 2 ] , and knowledge graph completion [ 16 ] . The rapid advancement of these models [ 8 ] , coupled with the growing availability of textual data and new KG construction methods [ 33 ] , has created a pressing need for benchmarks that can keep pace. However, existing benchmarks exhibit several important limitations. Early and widely used academic datasets such as Citeseer and Cora [ 24 ] are relatively small and consist of monorelational graphs, making them poorly suited for evaluating modern GNN architectures designed to operate on large-scale, heterogeneous, and multi-relational knowledge graphs. Even multirelational benchmarks such as AIFB [ 22 ] or domain-specific datasets in fields such as chemistry [ 32 ] and biology [ 34 ] represent entities primarily as symbolic identifiers or low-dimensional feature vectors, with little or no textual information. As a result, these benchmarks fail to capture the rich semantic content that is critical for many real-world applications. In contrast, large-scale knowledge graphs such as Wikidata [ 28 ] , YAGO [ 25 ] , DBpedia [ 3 ] , or domain-specific resources such as the Unified Medical Language System (UMLS) [ 5 ] contain extensive textual information associated with entities and relations. These resources are generally considered high-quality, as they are constructed from structured sources or curated by domain experts. While they provide valuable semantic information, they do not adequately reflect the challenges faced by graph learning models when operating on knowledge graphs that are automatically constructed from unstructured text. Recent advances in natural language processing have enabled the automatic construction of knowledge graphs directly from large collections of unstructured documents [ 13 , 1 , 19 ] . These text-derived knowledge graphs offer a more realistic representation of real-world data, as their nodes correspond to natural language terms extracted from text. However, such graphs are inherently noisy [ 17 , 6 ] . They often exhibit ambiguity, redundancy, fragmentation, and semantic inconsistencies introduced during the extraction process. Moreover, the resulting noise patterns are complex and domain-dependent, and therefore differ substantially from the simplified synthetic noise typically considered in controlled experimental studies [ 6 ] . Despite the growing importance of automatically constructed knowledge graphs, their evaluation remains challenging. Most existing studies focus primarily on the correctness of extracted triples, typically through manual annotation or automatic evaluation metrics [ 31 , 13 , 15 , 18 ] . However, such approaches generally assess only a subset of extracted relations and do not capture the overall quality of the resulting graph. Other works adopt a more practical perspective by evaluating the whole constructed graph through downstream tasks [ 19 , 11 , 30 ] . However, these approaches still suffer from an important limitation: the expected level of performance remains unclear. In particular, the absence of datasets that simultaneously provide (i) a textual corpus, (ii) automatically constructed knowledge graphs, and (iii) a high-quality reference graph makes it difficult to systematically analyze how extraction errors propagate and affect downstream learning performance [ 6 ] . As a result, it becomes challenging to determine whether performance degradation originates from the quality of the constructed graph itself or from the learning method applied on top of it. To bridge this gap, we introduce a novel dual-purpose benchmark designed to enable the controlled evaluation of both knowledge graph construction methods and GNN models. The benchmark is built on the MedMentions [ 20 ] biomedical corpus and includes two knowledge graphs automatically constructed from the same textual source using different extraction pipelines, together with a high-quality reference graph derived from the UMLS thesaurus. Because these graphs share a common set of entities and annotations, they allow direct comparison across graphs of varying quality under identical experimental conditions. The benchmark supports two complementary objectives. First, it enables the evaluation of knowledge graph construction methods by measuring how the quality of automatically extracted graphs affects performance on a downstream learning task (semi-supervised classification), compared to the performance obtained on the clean reference graph. This allows the performance loss caused by graph construction errors to be explicitly quantified. Second, it provides a standardized framework for evaluating the robustness of GNN models operating on noisy, text-derived knowledge graphs by comparing model performance across the automatically constructed graphs and the clean reference graph, which acts as an upper bound on the performance that can be achieved. To ensure reproducibility, the benchmark also includes a complete evaluation pipeline, providing standardized data loaders, training protocols, and baseline implementations of several representative GNN architectures. To summarize, the main contributions of this work are threefold: • A dual-purpose benchmark designed to jointly evaluate (i) knowledge graph construction methods through their impact on a downstream task and (ii) Graph Neural Network models operating on text-derived knowledge graphs under controlled experimental conditions. • The benchmark itself, which comprises a textual corpus, two automatically constructed KGs from different pipelines, and a high-quality reference graph derived from the UMLS Thesaurus. Importantly, all graphs share a common set of nodes annotated according to the same schema, enabling controlled comparisons across graphs of varying quality. • A reproducible and extensible evaluation framework, providing standardized data loaders, training protocols, and baseline GNN implementations, allowing fair and systematic benchmarking of both graph construction approaches and graph learning models. The remainder of this paper is structured as follows. Section