Understanding the Mechanisms Behind Structural Influences on Link Prediction: A Case Study on FB15k-237
Xiaobo Jiang, Yadong Deng · 2025
Read Understanding the Mechanisms Behind Structural Influences on Link Prediction: A Case Study on FB15k-237 by Xiaobo Jiang, Yadong Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01655.txt.
Introduction
Xiaobo Jiang and Yadong Deng investigate why link prediction models struggle on a popular knowledge graph benchmark, in a paper published in two thousand twenty-five. The benchmark, FB15k-237, removed inverse and symmetric triples to stop data leakage. Performance dropped sharply and progress since has been slow.
The authors blame unexplored structural effects. They design a sampling strategy that builds connected subgraphs with distinct structural features. Correlation and sensitivity analyses then run across several mainstream models. Relationship category distribution inside a subgraph mattered most. The size of strongly connected components came second. Follow-up analysis with the LIME explanation method deepens the picture. The study tells benchmark users which structural quirks most distort model scores.
Catalog record
| CREATOR | Xiaobo Jiang, Yadong Deng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01655
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding the Mechanisms Behind Structural Influences on Link Prediction: A Case Study on FB15k-237 Xiaobo Jiang, Yadong Deng Abstract FB15k-237 mitigates the data leakage issue by excluding inverse and symmetric relationship triples, however, this has led to substantial performance degradation and slow improvement progress. Traditional approaches demonstrate limited effectiveness on FB15k-237, primarily because the underlying mechanism by which structural features of the dataset influence model performance remains unexplored. To bridge this gap, we systematically investigate the impact mechanism of dataset structural features on link prediction performance. Firstly, we design a structured subgraph sampling strategy that ensures connectivity while constructing subgraphs with distinct structural features. Then, through correlation and sensitivity analyses conducted across several mainstream models, we observe that the distribution of relationship categories within subgraphs significantly affects performance, followed by the size of strongly connected components. Further exploration using the LIME model clarifies the intrinsic mechanism by which relationship categories influence link prediction performance, revealing that relationship categories primarily modulate the relative importance between entity embeddings and relationship embeddings and relationship embeddings, thereby affecting link prediction outcomes. These findings provide theoretical insights for addressing performance bottlenecks on FB15k-237, while the proposed analytical framework also offers methodological guidance for future studies dealing with structurally constrained datasets. 1 Introduction The evolution of modern artificial intelligence systems critically depends on high-quality datasets. Datasets not only determine models’ learning capabilities and predictive accuracy but also drive innovations in interdisciplinary research paradigms [ 1 , 2 , 3 , 4 , 5 , 6 ] . A prominent trend in dataset design involves eliminating redundant triples, such as inverse and symmetric relationships, to mitigate data leakage and facilitate genuine reasoning rather than simple memorization [ 7 ] . FB15k-237 is a typical example of this approach. Derived from the FB15K dataset [ 8 ] , FB15k-237 reduces data leakage by removing redundant relationship triples. However, its stringent construction criteria have led to notably lower performance across various models and slow performance improvements, as illustrated in Figure . Under the same model settings and training scale, the highest Mean Reciprocal Rank (MRR) achieved on FB15k-237 is only 0.42, significantly lower than those achieved on similar datasets such as FB15k (0.86) [ 9 ] and WN18RR (0.74) [ 10 ] . Consequently, FB15k-237 has increasingly been recognized as a more challenging benchmark for evaluation [ 11 ] . Although traditional research typically attributes performance gains to optimizing complex entity relationships, semantic associations, and model parameters, these methods show limited effectiveness on FB15k-237. For example, early translation-based models like TransE have been extended by more advanced variants such as TransD [ 12 ] and TransR [ 13 ] to better capture complex relationship patterns. Similarly, bilinear models including DistMult [ 14 ] , ComplEx [ 15 ] , and TuckER [ 16 ] utilize matrix and tensor decomposition to model semantic associations. Neural-network-based approaches, such as ConvE [ 17 ] , which leverages convolutional architectures, and HittER [ 18 ] , employing Transformer architectures, have also been extensively explored. Furthermore, graph neural network models like NBFNet [ 19 ] , as well as techniques involving hyperparameter tuning and diversified loss functions, have been investigated [ 20 , 21 , 22 ] . However, methods successful on conventional benchmarks fail to deliver expected performance improvements on FB15k-237. Performance curves across different datasets (shown in Figure 1 ) suggest that the traditional factors affecting model effectiveness do not entirely apply within its more restrictive structure. Figure 1: Curve of MRR performance changes across different datasets Since the construction of FB15k-237 only modified the dataset’s structure without changing the entity and relation information, the slow performance improvement may primarily relate to its structural features. We hypothesize that the fundamental performance bottleneck arises from an unexplored mechanism concerning how structural features of FB15k-237 impact model performance. Therefore, this study systematically investigates the mechanism through which dataset structural features influence link prediction performance on FB15k-237. To this end, we propose a randomized subgraph sampling method, utilizing a heuristic breadth-first search (BFS) strategy, to extract connected subgraphs of varying sizes and structural features from the large-scale multi-relational graph. Through correlation and sensitivity analyses conducted on these subgraphs, we quantify the influence of different structural features on model performance. Our results indicate that the distribution of relationship categories within subgraphs has the most significant impact, followed by the size of strongly connected components. Consistent observations across multiple models further verify the universality of these findings, confirming that these influences originate intrinsically from dataset properties rather than being model-specific. Finally, by employing the LIME model, we reveal the pathways through which relationship categories affect model performance. Specifically, we demonstrate that relationship categories primarily influence link prediction by modulating the relative importance of entity embeddings and relationship embeddings. In summary, this paper reveals the key structural features of the dataset that affect performance, quantifies their impact, and clarifies the pathways through which these features affect link prediction outcomes. Consistent results across multiple models further validate our findings. These insights provide a theoretical basis for improving performance on the FB15k-237 dataset, offering crucial support for future research aiming to enhance the learning capability of complex relationship categories, optimize resource allocation strategies, and improve embedding representations. Additionally, the analytical approach proposed in this study can be generalized to other datasets with performance bottlenecks, establishing a universal analytical framework for investigating and optimizing structural features. Consequently, this work contributes positively to the ongoing advancement and broader application of knowledge graph link prediction technologies. The structure of the paper is as follows: Chapter 1 introduces the research background and objectives of the study; Chapter 2 presents the relevant Knowledge Graph Embedding (KGE) models and previous studies on mechanism features; Chapter 3 discusses the experimental methods, including the random sampling method, the Gini index to measure data distribution imbalance, and the principles of using LIME for explanatory mechanism analysis; Chapter 4 is the experimental section, where the proposed ideas are validated; Chapter 5 summarizes the contributions of this work. 2 Related work 2.1 KGE model Knowledge graph completion (KGC) aims to identify missing interaction relationships between entities, thereby addressing the problem of incomplete knowledge graphs. Knowledge graph embedding (KGE) models infer new facts by mapping the elements of a knowledge graph into high-dimensional vectors and defining a scoring function. Typically, KGE models follow four steps [ 24 ] : (1) Random Initialization: Entities and relation vectors are randomly initialized; (2) Scoring Function: A scoring function is defined to measure the plausibility of a triple; (3) Interaction Mechanism: An interaction mechanism is designed to model the interactions between entities and relations and calculate the matching score of the triple; (4) Training Strategy: Strategies such as negative sampling and regularization are employed to maximize the confidence of triples and train the KGE model. Based on this process, various models have been proposed. The earliest of these is the translation-based TransE model, which is simple and efficient, but has limited capability in representing complex relationships. This limitation motivated subsequent research to further extend the modeling capacity of knowledge graph embeddings. For example, to address the shortcomings of TransE in handling complex relationship categories, researchers proposed TransH [ 25 ] and TransR [ 14 ] .