Transformer-Based Multimodal Knowledge Graph Completion with Link-Aware Contexts

Haodi Ma, Dzmitry Kasinets, Daisy Zhe Wang · 2025

Read Transformer-Based Multimodal Knowledge Graph Completion with Link-Aware Contexts by Haodi Ma, Dzmitry Kasinets, Daisy Zhe Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-15688.txt.

Introduction

Ma, Kasinets, and Wang present a study from twenty twenty-five on filling gaps in knowledge graphs that hold images and text. Such graphs store facts as links between entities, and missing links limit their use. Older methods need a vector for every entity, which bloats model size. Their method uses a vision-and-text model to turn images of an entity and its neighbors into text. A transformer model then treats link guessing as a text-to-text task. Fine-tuning with that added context drives the learning.

The approach shrinks model size sharply versus older embedding methods. It still matches rivals on several large test graphs. It needs little tuning of its settings to do so. The authors say borrowed vision context is a cheap way to enrich graph tasks.

Catalog record

CREATORHaodi Ma, Dzmitry Kasinets, Daisy Zhe Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-15688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transformer-Based Multimodal Knowledge Graph Completion with Link-Aware Contexts Haodi Ma, Dzmitry Kasinets, Daisy Zhe Wang Abstract Multimodal knowledge graph completion (MMKGC) aims to predict missing links in multimodal knowledge graphs (MMKGs) by leveraging information from various modalities alongside structural data. Existing MMKGC approaches primarily extend traditional knowledge graph embedding (KGE) models, which often require creating an embedding for every entity. This results in large model sizes and inefficiencies in integrating multimodal information, particularly for real-world graphs. Meanwhile, Transformer-based models have demonstrated competitive performance in knowledge graph completion (KGC). However, their focus on single-modal knowledge limits their capacity to utilize cross-modal information. Recently, Large vision-language models (VLMs) have shown potential in cross-modal tasks but are constrained by the high cost of training. In this work, we propose a novel approach that integrates Transformer-based KGE models with cross-modal context generated by pre-trained VLMs, thereby extending their applicability to MMKGC. Specifically, we employ a pre-trained VLM to transform relevant visual information from entities and their neighbors into textual sequences. We then frame KGC as a sequence-to-sequence task, fine-tuning the model with the generated cross-modal context. This simple yet effective method significantly reduces model size compared to traditional KGE approaches while achieving competitive performance across multiple large-scale datasets with minimal hyperparameter tuning. I Introduction Knowledge graphs (KGs) represent real-world information as structured triples (h,r,t) , where h and t are entities and r is a relation connecting them. This representation has proven valuable in various fields, including recommendation systems, question-answering, and drug discovery. KGs are widely used for organizing large-scale data and as auxiliary reasoning sources in systems like retrieval-augmented generation (RAG) [ 15 ] . While KGs traditionally store structural and textual information, their extension to incorporate multimodal data, such as images and videos, has led to the development of multimodal knowledge graphs (MMKGs). MMKGs bridge structured knowledge with unstructured content, associating entities with visual data to better ground their properties. Recent advances in natural language processing [ 19 ] and multimedia [ 8 ] have further highlighted the potential of MMKGs. However, MMKGs face two critical challenges: (1) Incompleteness, where missing links hinder downstream tasks, and (2) Cross-modal integration, where effectively connecting information across modalities remains non-trivial due to noise and inconsistencies in unstructured data. To address these challenges, multimodal knowledge graph completion (MMKGC) approaches aim to predict missing links in MMKGs by learning low-dimensional embeddings for entities and relations from visual, textual, and structural information. These methods use the embeddings to score candidate triples and identify missing links. Early works like IKRL [ 24 ] and TransAE [ 23 ] attempted to blend visual content into traditional knowledge graph embedding (KGE) models. IKRL integrated image features into KGE models by learning visual embeddings for entities, while TransAE introduced an auto-encoder module for multimodal fusion. More recent works, such as LAFA [ 17 ] and Mixture of Modality Knowledge Experts [ 26 ] , have improved multimodal fusion and reasoning in MMKGC. LAFA focuses on integrating multimodal data by considering link-aware fusion and neighborhood aggregation, demonstrating the potential of entity-specific multimodal reasoning. Despite these advances, existing MMKGC methods have several limitations: • Scalability of Embeddings: Traditional KGE and multimodal KGE models (e.g., TransE [ 1 ] , LAFA [ 17 ] ) require a unique embedding for each entity and relation, leading to linear scaling in both model size and inference time as the graph grows. • Uniform Treatment of Modalities: Many MMKGC models uniformly incorporate all associated images for an entity, ignoring their varying relevance to different links. While recent works like RSME [ 23 ] and MKGformer [ 2 ] consider image relevance, they focus on individual entities rather than the specific links between them. • Insufficient Use of Structural Context: Many existing MMKGC approaches focus on individual triples, neglecting the rich structural information provided by an entity’s neighbors. LAFA [ 17 ] partially addresses this issue with neighborhood aggregation but relies on KGE-based embeddings, which lack the flexibility and scalability as stated. Transformer-based models, such as KGT5 [ 16 ] , have emerged as promising alternatives for knowledge graph completion. These models frame link prediction as a sequence-to-sequence (seq2seq) task, where the model size is independent of the scale of the graph. Transformers also allow for straightforward incorporation of additional entity information without requiring complex model modifications. However, current Transformer-based models focus primarily on single-modal KGC, lacking explicit mechanisms to incorporate visual and relational contexts necessary for MMKGC. Furthermore, while vision-language models (VLMs) like BLIP [ 7 ] and InstructBLIP [ 5 ] provide powerful pretrained capabilities for visual-related tasks, fine-tuning them for MMKGC is computationally expensive and task-specific. Leveraging these VLMs for multimodal context generation without fine-tuning presents an efficient solution. Motivated by these challenges, we propose MMKG-T5, a Transformer-based MMKGC model designed to address the limitations of existing approaches. MMKG-T5 incorporates: • Link-Aware Multimodal Context: By leveraging a pretrained multi-image InstructBLIP T5 model [ 32 ] , our approach selects images based on the query link and generates link-aware textual descriptions from the images of entities and their neighbors. • Relation Context Enrichment: We enhance the model’s relational understanding by including context examples of the query relation from the training data. • Neighbor-Aware Aggregation: MMKG-T5 integrates information from an entity’s neighbors to fully utilize the structural properties of the MMKG. MMKG-T5 uses a Transformer-based seq2seq framework which enables dynamic context generation, reducing model size and improving scalability. By combining multimodal link context, relation context, and neighbor context, MMKG-T5 achieves competitive performance on large-scale MMKG datasets without the need for large-scale fine-tuning. The key contributions of this work are summarized as follows: • We propose MMKG-T5, a novel Transformer-based method for MMKGC, integrating multimodal, relational, and structural information. • We introduce a link-aware multimodal context generation approach that converts visual content into textual descriptions using a pretrained multi-image InstructBLIP T5 model. • We enrich the query with relation context examples and neighbor aggregation to improve relational and structural reasoning. • Our approach achieves competitive performance on large-scale MMKG datasets while maintaining computational efficiency, avoiding the need for large-scale fine-tuning of VLMs. II RELATED WORK II-A Knowledge Graph Completion Knowledge graph completion (KGC) aims to predict missing links in knowledge graphs by learning representations of entities and relations. Early approaches include embedding-based models like TransE [ 1 ] , which uses translation-based embeddings, and RotatE [ 18 ] , which models relations as rotations in a complex space. ComplEx [ 21 ] , as a representative of bilinear models, introduces a diagonal matrix with complex numbers to capture anti-symmetry. DistMult [ 27 ] is a bilinear embedding model for knowledge graph completion, scores triples using a dot product, while ComplEx [ 21 ] extends this by embedding entities and relations in a complex vector space to handle asymmetric and antisymmetric relations. These methods have successfully handled structural information but lack mechanisms to incorporate multimodal data. Recent advancements have explored the use of Transformers for KGC. KG-BERT [ 28 ] frames KGC as a binary classification task, using Transformer models to predict the plausibility of triples. KGT5 [ 16 ] extends this idea by treating KGC as a sequence-to-sequence (seq2seq) task, enabling the use of textual descriptions for entities and

Read the complete text