Elastic Weight Consolidation for Knowledge Graph Continual Learning: An Empirical Evaluation
Gaganpreet Jhajj, Fuhua Lin · 2025
Read Elastic Weight Consolidation for Knowledge Graph Continual Learning: An Empirical Evaluation by Gaganpreet Jhajj, Fuhua Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01890.txt.
Introduction
Knowledge graphs keep growing as new facts arrive, but the neural models trained on them tend to forget old facts when they learn new ones. Gaganpreet Jhajj and Fuhua Lin test a remedy in 2025: Elastic Weight Consolidation, a method that protects the weights a model relied on most for earlier tasks.
They apply the method to link prediction with TransE embeddings on the FB15k-237 benchmark, repeating the experiments across multiple random seeds. Their measurements show that the consolidation approach cuts the forgetting that plain sequential training suffers. The study gives practitioners a grounded sense of how far this regularization trick goes for ever-changing knowledge graphs.
Catalog record
| CREATOR | Gaganpreet Jhajj, Fuhua Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01890
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Elastic Weight Consolidation for Knowledge Graph Continual Learning: An Empirical Evaluation Gaganpreet Jhajj, Fuhua Lin Abstract Knowledge graphs (KGs) require continual updates as new information emerges, but neural embedding models suffer from catastrophic forgetting when learning new tasks sequentially. We evaluate Elastic Weight Consolidation (EWC), a regularization-based continual learning method, on KG link prediction using TransE embeddings on FB15k-237. Across multiple experiments with five random seeds, we find that EWC reduces catastrophic forgetting from 12.62% to 6.85%, a 45.7% reduction compared to naive sequential training. We observe that the task partitioning strategy affects the magnitude of forgetting: relation-based partitioning (grouping triples by relation type) exhibits 9.8 percentage points higher forgetting than randomly partitioned tasks (12.62% vs 2.81%), suggesting that task construction influences evaluation outcomes. While focused on a single embedding model and dataset, our results demonstrate that EWC effectively mitigates catastrophic forgetting in KG continual learning and highlight the importance of evaluation protocol design. [orcid=0000-0001-5817-0297, [email protected], ] *1 [orcid=0000-0002-5876-093X, [email protected], ] keywords Continual Learning ,Knowledge Graphs,Elastic Weight Consolidation ,Link Prediction † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: NORA’25: 1 st Workshop on Knowledge Graphs & Agentic Systems Interplay co-located with NeurIPS, Dec.1, 2025, Mexico City, Mexico † † address: School of Computing and Information Systems, Athabasca University, Canada 1 Introduction Knowledge graphs (KGs) depict structured information as networks of entities and their relations ( 22 ) , facilitating a variety of applications, including question answering ( 9 ) and recommendation systems ( 4 ) and educational systems ( 10 ; 12 ; 16 ; 8 ; 6 ; 5 ) . These real-world KGs evolve continuously as new information becomes available and existing knowledge is refined. Neural embedding models, such as TransE ( 1 ) , generate vector representations of entities and relations for link prediction. However, adapting these models to accommodate new information while retaining prior knowledge presents a significant challenge. Catastrophic forgetting occurs when neural networks, trained sequentially on multiple tasks, experience significant performance degradation on earlier tasks after learning new ones ( 18 ) . This phenomenon poses particular challenges for KG embeddings, where maintaining consistent representations across evolving information is essential. While continual learning methods have been developed for image classification and natural language processing, their effectiveness on KG link prediction remains underexplored. We investigate how Elastic Weight Consolidation (EWC) ( 14 ) , a regularization-based continual learning method, performs on KG link prediction. EWC protects important parameters learned in previous tasks by adding a quadratic penalty based on the Fisher Information Matrix, allowing networks to learn new tasks while preserving performance on old ones. We evaluate EWC on TransE embeddings using FB15k-237, a standard KG benchmark, and compare relation-based task partitioning (where triples are grouped by relation type) to random partitioning. Our experiments reveal that EWC substantially reduces catastrophic forgetting. On relation-based partitioned tasks, naive sequential training results in 12.62% forgetting (measured as MRR degradation from post-task performance), while EWC with regularization strength \lambda=10 reduces this to 6.85%, a 45.7% reduction. This demonstrates that regularization-based continual learning effectively preserves KG embeddings across sequential tasks. We also observe that the task partitioning strategy significantly affects the measured forgetting. Naive sequential training on relation-based tasks exhibits 12.62% forgetting, compared to only 2.81% on randomly partitioned tasks, a 9.8%point difference. This suggests that evaluation protocols, particularly how tasks are constructed from datasets, influence continual learning metrics and should be carefully explored when it comes time to design the study. Our study focuses on TransE embeddings for the FB15k-237 dataset across four relation-based tasks. While this scope limits generalizability, it provides rigorous evidence that EWC reduces catastrophic forgetting in KG continual learning and raises essential questions about task construction in continual learning evaluation. This work addresses a critical challenge for knowledge graph-based AI agents: maintaining knowledge representations as new information arrives. As agents increasingly rely on KG-based memory and reasoning, the ability to incorporate new knowledge while preserving existing facts becomes essential for long-term autonomous operation. 2 Related Work KG Embeddings. TransE ( 1 ) represents relations as translations in embedding space, learning vectors such that \mathbf{h}+\mathbf{r}\approx\mathbf{t} for true triples (h,r,t) . Extensions include TransH ( 28 ) , RotatE ( 23 ) , and ComplEx ( 26 ) . Continual Learning. Methods for mitigating catastrophic forgetting include regularization approaches like EWC ( 14 ) and Learning without Forgetting ( 15 ) , replay-based methods that store and revisit previous examples ( 20 ) , and architectural approaches that allocate separate parameters for different tasks ( 21 ) . EWC estimates parameter importance using the Fisher Information Matrix and adds regularization penalties to protect important weights during subsequent task training. KG Continual Learning. Prior work has explored continual learning for KGs across various contexts. 27 studied multi-task feature learning for KG-enhanced recommendation. 2 demonstrated continual learning methods for KG embeddings in robotic manipulation tasks, evaluating multiple architectures including TransE, DistMult, and ComplEx. 29 introduced the PS-CKGE benchmark, focusing on pattern shifts and demonstrating that these shifts exacerbate catastrophic forgetting beyond what would be expected from simple data scaling. Recent work has also examined embedding adaptation ( 3 ) and temporal KGs. While these comprehensive benchmarking efforts establish the landscape of KG continual learning, a focused empirical analysis of classic regularization methods like EWC, with explicit investigation of task partitioning effects, remains limited, motivating our study. 3 Methodology 3.1 Problem Formulation A KG is \mathcal{G}=(\mathcal{E},\mathcal{R},\mathcal{T}) where \mathcal{E} is the set of entities, \mathcal{R} is the set of relations, and \mathcal{T}\subseteq\mathcal{E}\times\mathcal{R}\times\mathcal{E} is the set of true triples. TransE learns embeddings \mathbf{h},\mathbf{r},\mathbf{t}\in\mathbb{R}^{d} by minimizing: \mathcal{L}=\sum_{(h,r,t)\in\mathcal{T}}\sum_{(h^{\prime},r,t^{\prime})\in\mathcal{T}^{\prime}}\max(0,\gamma+d(\mathbf{h}+\mathbf{r},\mathbf{t})-d(\mathbf{h}^{\prime}+\mathbf{r},\mathbf{t}^{\prime})) (1) where \mathcal{T}^{\prime} contains negative samples, d(\cdot,\cdot) is L2 distance, and \gamma is the margin. In continual learning, we partition \mathcal{G} into tasks \mathcal{G}_{1},\ldots,\mathcal{G}_{T} and train sequentially. After training on task i , we measure performance M_{i}^{j} on task j\leq i . Forgetting for task j after learning task i is: F_{i}^{j}=M_{j}^{j}-M_{i}^{j}\quad\text{for }i>j (2) We report average forgetting at the end of training: \bar{F}=\frac{1}{T-1}\sum_{j=1}^{T-1}F_{T}^{j} (3) 3.2 Elastic Weight Consolidation EWC protects important parameters by adding a regularization term to the loss when training on task i : \mathcal{L}_{\text{EWC}}^{i}=\mathcal{L}^{i}+\frac{\lambda}{2}\sum_{k}F_{k}(\theta_{k}-\theta_{k,i-1}^{*})^{2} (4) where \theta_{k,i-1}^{*} are optimal parameters after task i-1 , F_{k} is the Fisher Information diagonal approximation, and \lambda controls regularization strength. The Fisher Information Matrix diagonal is: F_{k}=\mathbb{E}_{(h,r,t)\sim\mathcal{G}_{i-1}}\left[\left(\frac{\partial\log p(y|x;\theta)}{\partial\theta_{k}}\right)^{2}\right] (5) We compute this empirically using all triples from the previous task, processed in mini-batches (see Appendix A for implementation details). 3.3 Task Partitioning We evaluate two task partitioning strategies to assess how task construction affects catastrophic forgetting: Relation-based partitioning: We partition FB15k-237 by grouping all triples that share the same relations. To create balanced task sizes, we sort the 237 relations by frequency (number of triples per relation) and assign them to four tasks in round-robin order: the most frequent relation goes to Task 1, the second-most to Task 2, the third-most to Task 3, the fourth-most to Task 4, the fifth-most back to Task 1, and so forth. This ensures each task receives approximately 59 relations with a mixture of common and rare relation types, while maintaining relation-level coherence —all triples with the same relation