MERGE: A Modal Equilibrium Relational Graph Framework for Multi-Modal Knowledge Graph Completion Authors: Yuying Shang, Kun Fu, Zequn Zhang, Li Jin, Zinan Liu, Shensi Wang, Shuchao Li, Stefanos Kollias Published: 2024 ABSTRACT AbstractThe multi-modal knowledge graph completion (MMKGC) task aims to automatically mine the missing factual knowledge from the existing multi-modal knowledge graphs (MMKGs), which is crucial in advancing cross-modal learning and reasoning. However, few methods consider the adverse effects caused by different missing modal information in the model learning process. To address the above challenges, we innovatively propose a Modal Equilibrium Relational Graph framEwork, called MERGE. By constructing three modal-specific directed relational graph attention networks, MERGE can implicitly represent missing modal information for entities by aggregating the modal embeddings from neighboring nodes. Subsequently, a fusion approach based on low-rank tensor decomposition is adopted to align multiple modal features in both the explicit structural level and the implicit semantic level, utilizing the structural information inherent in the original knowledge graphs, which enhances the interpretability of the fused features. Furthermore, we introduce a novel interpolation re-ranking strategy to adjust the importance of modalities during inference while preserving the semantic integrity of each modality. The proposed framework has been validated on four publicly available datasets, and the experimental results have demonstrated the effectiveness and robustness of our method in the MMKGC task.Keywords: multi-modal knowledge graph, knowledge graph representation, graph attention network, information integration 1. Introduction A knowledge graph, such as Wikidata [1], Freebase [2], and WordNet [3], is regarded as a complex semantic network that typically stores vast amounts of factual information in the form of (subjectentity,relation,objectentity) or (es,r,eo) for short. In recent years, the emergence of large language models (LLMs) has achieved impressive results in natural language processing (NLP). However, due to limited factual knowledge, LLMs often suffer from hallucinations, generating incorrect factual statements. Given the advantage of knowledge graphs in capturing the intricate representation of entities and their relationships, integrating the knowledge graph with LLMs presents a potentially reasonable solution to overcome these challenges. Nevertheless, due to the difficulty and cost of knowledge acquisition, existing knowledge graphs often contain invalid triplets with missing entities or relations, leading to sparsity and incompleteness. Therefore, to continuously improve the construction of knowledge graphs, researchers have proposed the task of knowledge graph completion (KGC) to promote knowledge discovery. Traditional knowledge graph completion methods such as TransE [4] primarily consider the structural characteristics between entities. These methods measure the validity of each triplet through knowledge graph embedding techniques, optimizing scoring functions to obtain entity and relation representations. The real world consists of a variety of modal information [5], where entity description enhances the understanding of abstract concepts, and additional image information visually aids in distinguishing entity attributes and categories. To bridge the information gap in traditional knowledge graphs, researchers have collected and enriched the entity with textual and image information, extending traditional knowledge graphs to multi-modal knowledge graphs. Nowadays, multi-modal knowledge graphs have been widely utilized in various knowledge-driven downstream tasks, such as object detection [6,7], fake detection [8,9], and sentiment analysis [10,11]. In order to address the sparsity in multi-modal knowledge graphs, researchers have proposed the task of multi-modal knowledge graph completion (MMKGC). Figure 1 provides an illustration of the MMKG in the film domain, extracted from the FB15K [12] dataset. Referring to the figure, each entity is accompanied by textual descriptions and multiple images. The entity “James Cameron” and “Titanic” are connected by the relation “Director” and “Produced by”, which can be rewritten as (JamesCameron,Director,Titanic) and (Titanic,Producedby,JamesCameron) for their interrelationship representations. An example of multi-modal knowledge graph in the film domain. The visual information of the entity “Titanic” can be represented by the movie posters and stills, while its textual information can be represented by sentence-level descriptions. Each entity is equipped with text descriptions and multiple images. In recent years, a series of methods have been proposed for the MMKGC task. Based on the information encoding and fusion approaches, we categorize the methods into single-stream frameworks and multi-stream frameworks. The early MMKGC methods utilize single-stream frameworks, which tightly integrate diverse modal features of an object entity into a single embedding vector before a self-defined encoder. Then, a decoder based on the scoring function of the traditional knowledge representation models was used to reconstruct the encoded features. TransAE [13] uses a five-layer multi-modal encoder to automatically fuse visual and textual features, applying a TransE decoder for entity and relation prediction. We present the illustration of the single-stream frameworks in Figure 2a. Additionally, some researchers have proposed multi-stream frameworks to encode the information of each modality separately and exchange the information through a self-defined interaction module. The process is shown in Figure 2b. IMF [14] proposes a two-stage fusion model, which encodes each modality by a modal-specific encoder, proposing a decision fusion module to better integrate the complementary of different modalities. Illustration of the difference between single-stream frameworks and multi-stream frameworks. (a) depicts the process of single-stream frameworks, which integrate diverse modal features into an ensemble embedding for the object entity. (b) depicts the process of multi-stream frameworks, which encode the information of each modality separately, integrating the multi-modal features through a self-defined interaction module. However, the single-stream frameworks fail to retain the unique characteristics of each modality, and the representational capacity of each modal is affected by the features of other modalities, resulting in great information loss and overfitting issues [15]. On the other hand, multi-stream frameworks have not sufficiently considered the noise impact of the encoded missing modal information during the interaction process, which results in inadequate interpretability of the output entity features. Figure 3 shows the modality information distribution of 14,951 entities in the FB15K dataset. It is obvious to see that visual information is more challenging to collect compared to textual information. Furthermore, the number of entities decreases as the completeness of their assigned modal attributes increases. In fact, there is still a lack of effective discussion on how to effectively address the incomplete multi-modal information issue of entities. The modality information distribution of 14,951 entities in FB15K dataset. As a heterogeneous graph containing various attributions of nodes and relationships, the multi-modal knowledge graph requires a relation-specific encoding method to achieve precise relational embeddings. Graph neural networks (GNNs) [16] are proposed as a specialized encoding method for graph-structured data to integrate topological structure information and attribute features. By using weighted summation of neighboring nodes to update the hidden states of the object nodes, GNNs can fully exploit the potential connections between nodes and their surrounding objects, which is particularly suitable for obtaining implicit representations of missing modal information in MMKGs. Some researchers have attempted to adopt GNNs into the MMKGC task. Among these efforts, HRGAT [17] stands out. As a single-stream framework, it constructs hyper-nodes with visual, textual, and numerical features, establishing a relational graph attention network to update the representations of hyper-nodes. Nevertheless, HRGAT does not consider the specialized expression among modal features, and thereby, the importance of each modality in representing the object entities has not been adequately quantified. Inspired by the existing multi-stream frameworks and the graph neural networks, we innovatively propose MERGE, which facilitates multi-modal information interaction while preserving the specific semantic representation of each modality. Figure 4 is the graphical abstract of this paper. To accurately capture the multi-modal information of entities, we first utilize a visual filtering gate to filter the noise images, pre-training the visual and textual features on the CLIP [18] model to obtain semantically aligned modal-specific entity embeddings. Then, we propose three directed relational graph attention networks (DR-GATs) for the structural, visual, and textual modality based on the origin graph structure information of the multi-modal knowledge graphs. By employing the entity–relation combination operations and relation-aware message passing functions, our framework achieves the implicit representation of missing modal features for each entity. The proposed modal-specific DR-GAT can not only align image, text description, and structural information at the entity level but also enhances the interpretability of the entities’ representation embeddings. Subsequently, we adopt decomposition factors in the multi-stream interaction fusion module to capture intra-modal and inter-modal information interaction. Finally, we propose an interpolation re-ranking strategy to reorder the predicted results based on the importance of each modality for the MMKGC task. The graphic abstract of the article. The blue and green squares stand for limitations and causes, respectively, while the orange squares stand for the solutions of MERGE. We validate the MERGE on FB15K-237-IMG, WN18RR-IMG, DB15K-IMG, and MKG-W datasets, and the results have shown that the proposed framework generally outperforms the baseline models, achieving improvements of 0.5%, 1.9%, 5.8%, and 0.6% in the MRR metric, proving its correctness and robustness. The conducted ablation experiments also demonstrate the effectiveness of each module in our approach. Additionally, we implement a contrast experiment to validate the superiority of the multi-stream framework over the single-stream framework in addressing the MMKGC task based on our proposed DR-GAT network. In conclusion, we summarize the contributions of this paper as follows: We explore a method to address the issue of missing modal information among MMKGs. The experimental results demonstrate that the multi-stream interaction strategy outperforms the single-stream fusion strategy when facing the incomplete multi-modal corpora issue. We explore a method to address the issue of missing modal information among MMKGs. The experimental results demonstrate that the multi-stream interaction strategy outperforms the single-stream fusion strategy when facing the incomplete multi-modal corpora issue. We propose a Modal Equilibrium Relational Graph framEwork for multi-modal knowledge graph completion, named MERGE. This framework achieves modality alignment at both the explicit structural level and the implicit semantic level for structural, textual, and visual features, which preserves the modal-specific information and enhances the interpretability of the represented embeddings. We propose a Modal Equilibrium Relational Graph framEwork for multi-modal knowledge graph completion, named MERGE. This framework achieves modality alignment at both the explicit structural level and the implicit semantic level for structural, textual, and visual features, which preserves the modal-specific information and enhances the interpretability of the represented embeddings. Experimental results on four publicly available datasets demonstrate that MERGE outperforms state-of-the-art approaches in the MMKGC task. Experimental results on four publicly available datasets demonstrate that MERGE outperforms state-of-the-art approaches in the MMKGC task. 2. Related Works Knowledge representation learning essentially converts entities and relationships in a knowledge base into low-dimensional, dense vectors. In this section, we provide an overview of approaches related to our framework. 2.1. Knowledge Graph Embedding Various knowledge graph embedding (KGE) methods have been proposed for the knowledge graph completion (KGC) task, which define scoring functions to project entities and relations into vector space and evaluate the probabilities of valid triplets. We categorize the KGE methods into the following four types: Translation-based models: TransE [4] is regarded as the traditional translation model, known for its simplicity and efficiency, which considers the object entities as the transformation from subject entities via a relation vector. Subsequently, Lin et al. [19] proposed TransR to project entities and relations into different semantic spaces, providing varying weights within different relations. In contrast to the traditional methods that project entity embeddings into the Euclidean space, RotatE [20] defines each object entity as a rotation from the subject entity in the complex space, demonstrating good performance in inferring various relational patterns. Yu et al. [21] proposed TransEllipsoid and TransCuboid to embed each concept as an n-dimensional ellipsoid or cuboid to establish the anisotropy of concept embeddings. Although the translation-based KGE models are popular for their simplicity, they fall short in capturing entities and relationships with complex semantics, which are prone to confusing the semantic representations of similar entities. Bilinear models: The bilinear models represent the relationships between entities by three-dimensional tensors. RESCAL [22] introduces a matrix-based tensor decomposition method that performs representation learning by minimizing the reconstruction error of the predicted triplets. DistMult [23] adopts a bilinear neural network to model multi-relational graphs, simplifying calculations and preventing the overfitting issue of the model. ComplEx [24] extends the representation space to the complex space, showing advantages in modeling asymmetric relationships. These approaches demonstrate superiority in expressing complex relationships; however, the operations on high-dimensional tensors reduce the training and inference efficiency. Neural network-based models: The emergence of deep neural networks has facilitated the utilization of convolutional neural networks (CNNs) in knowledge graph representation. ConvE [25] first leverages convolutional and fully connected layers to facilitate interactions between entities and corresponding relations. CrossE [26] introduces an interaction matrix to model the crossover interactions between entities. InteractE [27] employs feature permutation, checkered reshaping, and circular convolution to capture diverse heterogeneous interactions, enhancing the model’s expression ability. The deep neural networks have demonstrated superior performance on some knowledge graph datasets due to their effectiveness in parameterization. However, these approaches do not fully utilize the structural features of knowledge graphs and their output embeddings lack interpretability. GNN-based models: Traditional deep learning methods have limited performance when dealing with non-Euclidean data. Therefore, the graph neural networks have been proposed to aggregate entity features for the KGC task by leveraging the structural information of knowledge graphs. GCN [28] first proposes the graph convolution method to aggregate information from neighboring entities. R-GCN [29] associates the weight matrix with different types of relations to account for the influence of relationships between entities. However, the R-GCN method only establishes a relation embedding matrix without incorporating the relation embeddings into the update process. Then, CompGCN [30] proposes to utilize an entity–relation composition operation to learn the joint embeddings of entities and relations. SR-GNN [31] develops two semantic aggregation modules to pay attention to the influence of semantic information among neighboring entities. Nevertheless, with the continuous enrichment of knowledge graph information, the aforementioned KGE methods are no longer sufficient to deal with the growing complexity of the data demands. These approaches primarily focus on handling structural fact triplets, neglecting the potential value of multi-modal information. 2.2. Multi-Modal Knowledge Graph Representation The mainstream methods for MMKGC tasks typically extend traditional KGE models with diverse modal features. Based on different multi-modal information interaction strategies, we classify the multi-modal knowledge graph representation methods into single-stream frameworks and multi-stream frameworks: Single-stream frameworks: These methods typically integrate various modal features of object entities into a single vector, considering the semantic space of the fused embeddings as the representation space. IKRL [32] is the first to introduce the image embeddings into the TransE scoring function, utilizing the visual information of entities. TransAE [13] further integrates image and description features of entities, extending TransE into a multi-modal scoring function to decode the predicted triplets. Subsequently, KBLRN [33] considers expanding the available modal information by introducing numerical features for entity representations. However, substantial noise is contained in image copra in MMKGs, which may interfere with the effectiveness of the information fusion. To address this, RSME [34] proposes a filter gate and a forget gate to obtain valuable visual embeddings, modeling entity relationships through the ComplEx scoring function. Considering the limited performance of translation-based models in handling complex relationships, KMAGCN [35] adopts the graph neural network to model the structural information of the knowledge graph, integrating textual information, visual information, and knowledge concepts into a unified framework to capture semantic representations of entities. HRGAT [17] further incorporates numerical features into a graph attention network, considering the varying importance of neighboring nodes. CMGNN [36] aggregates information from local neighboring entities under a graph neural network framework with a contrastive learning mechanism. Although the single-stream frameworks have achieved superior performance on the MMKGC task, few consider the specialized expression among modal features, resulting in information loss and model overfitting issues. Multi-stream frameworks: These methods define different modal encoders to first learn the unique features of each modality, and then leverages customized information propagation modules for the interaction of multi-modal features. OTKGE [37] models the multi-modal fusion process as a transportation process that moves different modal embeddings into a unified space, considering the optimal transport strategy for three modalities. However, this method primarily focuses on binary relations, whereas real-world multi-modal information often manifests as beyond-binary relations. IMF [14] utilizes the independent representation of different modalities and applies the contrastive learning method as an additional constraint, providing learnable weight parameters for all modal predictions in the decoding module to better integrate the complementary of different modalities. However, the contextual information for each triplet is represented solely through a bilinear outer product between entity and relation embeddings. Both of the aforementioned approaches oversimplify relationships in the real world, failing to account for the semantic associations between neighboring entities and the target entity. In fact, incorporating relation embeddings into the GCN formulation enables more sophisticated modeling of real-world relationships. MMRNS [38] introduces a knowledge-guided cross-modal attention module, constructing a contrastive semantic sampler to enhance the semantic representation of multi-modal entities. Since this method tightly couples textual and visual features together, its scalability is limited. Therefore, the absence of features from one modality can result in a significant degradation in the method’s performance. Nevertheless, modality completeness cannot always be guaranteed in real-world scenarios. Some researchers have considered introducing generative models such as GAN [39] and VAE [40] to assist in completing modality information or generating labeled data. Zhang et al. [41] proposed MACO to generate missing visual features, designing cross-modal contrastive loss to improve the quality of the generated features. Ref. [42] introduced a weighted semi-supervised DNN-based approach and a GAN-based method to address challenges of limited labeled data and missing unlabeled data in indoor localization. However, assigning values to missing values through specific data may introduce additional noise during the fusion process. Some other methods have considered joint learning approaches. Han et al. [43] proposed a joint training model consisting of two modality-specific encoders and a shared classifier, which implicitly integrates information from different modalities for emotion recognition. From our perspective, such implicit fusion methods are less effective at providing clear insights into the contributions of each modality to the final decision. In contrast, decision-level fusion has been shown to offer greater interpretability and flexibility by combining modality-specific outputs explicitly [44,45]. Therefore, we propose MERGE, a multi-stream framework-based approach, which applies three modal-specific directed relational graph attention networks to fully retain the unique features of each modality, while modeling the complex relationships through the graphical information of knowledge graphs. Moreover, we propose an interpolation re-ranking strategy, leveraging the decision-level fusion of modality-specific decoding scores to mitigate the impact of missing modality data on model performance. Table 1 presents the comparison between MERGE and other existing knowledge graph representation methods. Table 1The comparison between MERGE and other existing knowledge graph representation methods. “Agg.” denotes the aggregation process, “Import.” denotes the calculation of the importance scores, and “Info.” denotes the information.ModelsEntity Agg.Relation Agg.Neigh-Boring Import.Multi-Modal Info.Modal-Specific Info.GCN [28]✓××××CompGCN [30]✓✓×××SR-GNN [31]✓✓✓××HRGAT [17]✓✓✓✓×IMF [14]×××✓✓MERGE✓✓✓✓✓ The comparison between MERGE and other existing knowledge graph representation methods. “Agg.” denotes the aggregation process, “Import.” denotes the calculation of the importance scores, and “Info.” denotes the information. 3. The Proposed Method 3.1. Preliminary In this section, we introduce the fundamental concepts utilized in this article and present the definition of the MMKGC task. The knowledge graph can be defined as G={E,R,T}, where E,R, and T denote the set of entities, relations, and known triplets, respectively. Each entity E has three types of information in a multi-modal knowledge graph, including structural information Es, textual information Et, and visual information Ev. Triplet T={(s,r,o)} in an MMKG can be interpreted as the connection between the subject entity s and object entity o via the relation r. Following the previous works [30,46], we add the inverse relations Rinv = {r−1∣r∈R} to the relation set to enable bidirectional information transmission in the knowledge graph. The set of entities, relations, and triplets of an incomplete multi-modal knowledge graph G′={E′,R′,T′} can be defined as: where ⊤ is regarded as the self-loop relations, ensuring that the information of entity nodes themselves will not be lost during feature updates. The multi-modal knowledge graph completion task aims to predict the missing head or tail entities in triplets based on the known multi-modal entity set E′ and relation set R′, which can be defined as: To simplify the expression of the MMKGC task, we divide it into the forms of the head entity prediction (?,r,t) and tail entity prediction (h,r,?). Figure 5 illustrates an example of predicting the tail entity in a triplet (JaneFonda,filmactor,?). By taking into account the multi-modal information of all entities and relations in the MMKG, the model will select the entity of the film named Julia that matches the given relationship to accomplish the incomplete triplet. The illustration of the example of tail entity prediction in the MMKGC task, which can be formed as (h,r,?). 3.2. Framework The proposed MERGE adopts an multi-stream encoder–decoder architecture. The overall architecture of MERGE is depicted in Figure 6, which includes four components: modal information extraction, missing information capturing, multi-stream interaction fusion, and interpolation re-ranking strategy. Our framework first utilizes the modal information extraction module to filter and obtain aligned textual and visual features of entities. Then, by defining a directed relational graph attention network for each modality, it produces the implicit representation of the entities’ missing modal features, aligning the structural, textual, and visual features at the entity level. Subsequently, the MERGE employs a low-rank modal fusion method to facilitate intra-modal and inter-modal information interaction. Finally, we propose an interpolation re-ranking strategy to reorder the prediction results based on the importance of each modality while preserving the unique semantics of each modality. The specific functions and processing steps of each part will be explained in detail in the following sections. The framework of MERGE for multi-modal knowledge graph completion. 3.2.1. Modal Information Extraction The existing MMKGs consist of entities with various modalities; each represents a different scenario. Therefore, it is crucial to effectively capture information from each modality and address the semantic alignment of the entities’ image–text information. As a textual-visual pre-trained model, CLIP [18] has demonstrated promising zero-shot performance on various image classification datasets. Through the contrastive learning strategy, CLIP generates representations by maximizing the cosine similarity between the original text and its corresponding image. Thus, to obtain the matching visual and textual features of entities, we first pre-train the image–text pairs of entities based on CLIP. As the Vision Transformer (ViT) [47] can capture more global features than the traditional convolutional neural networks, we adopt the CLIP architecture based on ViT to extract multi-modal features of entities. To be more specific, each entity is equipped with n text descriptions D={d1,d2,d3,…,dn} and m corresponding images I={i1,i2,i3,…,im}. We feed the images into the visual side and the description sequences into the textual side of CLIP, saving the pre-trained model parameters by optimizing the contrastive loss function for further feature extraction. Textual Feature Extraction: To extract entities’ textual description features, we apply the pre-trained textual side of CLIP as our sentence-level feature encoder. The text encoder employs a 12-layer Transformer block of 512 width with eight multi-head self-attention units. Raw texts are transformed into tokens by byte pair encoding [48] with a vocabulary size of 49,152. The input sequence length is limited to 128 and a special [CLS] token is added before as a global aggregation token. The resulting 512-dimensional embedding of the [CLS] token is served as the textual embedding of the entity. Visual Feature Extraction: The image corpora of a practical MMKG contains a large amount of noise images due to the high cost and difficulty of acquisition, which may mislead the model to perform an incorrect inference in the MMKGC task. The noise mainly originates from two sources: incorrect images crawled from the internet and images with limited visual semantic similarity to the related entity. According to the above reasons, it is a challenging problem to filter and choose a typical visual representation of the entity. Inspired by the method of [34], we introduce a visual filtering gate before the image encoder to eliminate noisy images from the datasets. Figure 7 is an illustration of the visual filtering gate. Specifically, based on the perceptual hash algorithm (pHash) [49], we first convert the color image to grayscale and resize it to a fixed dimension of 32 × 32. Then, a Discrete Cosine Transform (DCT) [50] is applied to extract the primary features of the image. From the DCT results, we select the top-left 8 × 8 submatrix and generate a 64-bit perceptual hash value based on the average value of this region. Image similarity is then assessed by calculating the Hamming distance [51] between hash values, with a smaller Hamming distance indicating greater similarity. The illustration of the visual filtering gate, which is proposed to select the typical representative image of each entity. For a given entity e, we can define its image corpora as Ie={img1,img2,img3,…,imgm}. The proposed visual filtering gate utilizes the pHash value of the images to identify the typical representative image imge of the entity by similarity calculating function S(·): The CLIP visual encoder is established based on “ViT-B/32”, which consists of 12 Transformer blocks and 12 multi-head self-attention units. Images are first converted into patches and a position encoding is added with the encoded image tokens. We also add a [CLS] token at the beginning of the input tokens. After passing through a global pooling function, the 512-dimensional vector of [CLS] is achieved as the visual embedding of the entity. Structural Feature Extraction: To better preserve the graphical structure information in MMKGs, we regard each entity as a graph node and the relations between entities as edges. The Xavier initialization method is employed to initialize the node and edge features with d0 dimensions. After obtaining the semantically aligned image–text entity features, we project the textual and visual embeddings into the same semantic space as the structural embeddings by trainable weights Wt∈Rdt×d0 and Wv∈Rdv×d0, respectively. Here, dt and dv are the dimensions of the extracted textual and visual embeddings. The missing modal features in entities will be initialized by zero vectors. We adopt the modal-specific directed relational graph attention network mentioned in Section 3.2.2 to update the implicit representation of the entities’ missing modal information. 3.2.2. Missing Information Capturing To retain the unique semantic information of each modality, we adopt a multi-stream framework that constructs a directed relational graph attention network (DR-GAT) for each modality. Algorithm 1 presents the pseudocode of the DR-GAT process. Section 4.5.7 qualitatively analyzes the effectiveness of the multi-stream framework compared with the single-stream framework. The construction of DR-GAT will be introduced in detail in the following content. According to the graph neural networks, entities are treated as nodes, and relations are regarded as edges. To update the implicit representations of entities in multi-modal knowledge graphs, we employ a message passing function that aggregates information from neighboring nodes through edges. Unlike the traditional GCN-based [28,52] methods, our model utilizes an entity–relation combination operation in the message passing function for information aggregation. Additionally, we employ a relation-aware attention mechanism to capture the importance of neighbors through relational correlation patterns. Algorithm 1 The aggregation process of DR-GATInput: Modal-specific embeddings in triplet (s,r,o)Output: Entity and relation embedding e and r 1:for L∈(1,2,…,k−1) do ▹The layer number of DR-GAT is L 2: ϕ(es,er)←Wr∘es+Wr∘es∘er 3: m(s,r,o)←Wdirϕ(es,er) 4: for j∈N(s) do ▹N(s) is the neighboring set of entity s 5: for ha∈(1,2,…,H) do ▹H is the number of attention head 6: attj←σ(Wattm(es,er,ej)),j∈N(s) 7: αj←softmax(attj) ▹αj is the attention coefficient of node j 8: end for 9: eoj←1H∑h=1Hαjhmh(es,er,ej)10: end for11: eok←σ∑j∈N(s)eojk−112: erk←Wrk−1erk−113:end for14:return e, r Input: Modal-specific embeddings in triplet (s,r,o) Modal-specific embeddings in triplet (s,r,o) Output: Entity and relation embedding e and r Entity and relation embedding e and r 1:for L∈(1,2,…,k−1) do ▹The layer number of DR-GAT is L for L∈(1,2,…,k−1) do ▹The layer number of DR-GAT is L 2: ϕ(es,er)←Wr∘es+Wr∘es∘er ϕ(es,er)←Wr∘es+Wr∘es∘er 3: m(s,r,o)←Wdirϕ(es,er) m(s,r,o)←Wdirϕ(es,er) 4: for j∈N(s) do ▹N(s) is the neighboring set of entity s for j∈N(s) do ▹N(s) is the neighboring set of entity s 5: for ha∈(1,2,…,H) do ▹H is the number of attention head for ha∈(1,2,…,H) do ▹H is the number of attention head 6: attj←σ(Wattm(es,er,ej)),j∈N(s) attj←σ(Wattm(es,er,ej)),j∈N(s) 7: αj←softmax(attj) ▹αj is the attention coefficient of node j αj←softmax(attj) ▹αj is the attention coefficient of node j 8: end for end for 9: eoj←1H∑h=1Hαjhmh(es,er,ej) eoj←1H∑h=1Hαjhmh(es,er,ej) 10: end for end for 11: eok←σ∑j∈N(s)eojk−1 eok←σ∑j∈N(s)eojk−1 12: erk←Wrk−1erk−1 erk←Wrk−1erk−1 13:end for end for 14:return e, r return e, r Entity–Relation Combining Operation: Graph neural networks aggregate information from neighboring nodes to the object node through relation edges. Studies in [30] have shown that relations can provide relevant information for entity representation during the information aggregation process. Therefore, we introduce an entity–relation combination operation, which can be defined as follows: where ϕ is denoted as the fusion operator, and es,er,eo∈Rd0 is the corresponding embedding of the subject entity, relation, and object entity. Since the DR-GAT is constructed for each modality, e is represent as the textual, visual, and structural embeddings. Inspired by TransE [4], DistMult [23], HolE [53], and CrossE [26], we propose four entity–relation combining operations for the combination of entity and relation information. The origin definitions of their score functions are shown in Table 2. Each entity–relation combining operation is defined as follows: •Subtraction (Sub): ϕ(es,er)=es−er; Subtraction (Sub): ϕ(es,er)=es−er; •Multiplication (Mult): ϕ(es,er)=es∘er; Multiplication (Mult): ϕ(es,er)=es∘er; •Circular-correlation (Corr): ϕ(es,er)=es★er; Circular-correlation (Corr): ϕ(es,er)=es★er; •Crossover interaction (Cross): ϕ(es,er)=Wr∘es+Wr∘es∘er. Crossover interaction (Cross): ϕ(es,er)=Wr∘es+Wr∘es∘er. Table 2The origin scoring function φ of different knowledge graph embedding models. Symbol ∘ denotes Hadamard product, ★ denotes circular convolution, σ is a non-linear activation function.MethodScore FunctionTransE ∥es+er−eo∥p DistMult 〈es,er,eo〉 HoLE σ(erT(es★eo)) CrossE σ(eoTtanh(wr∘es+wr∘es∘er)) The origin scoring function φ of different knowledge graph embedding models. Symbol ∘ denotes Hadamard product, ★ denotes circular convolution, σ is a non-linear activation function. Symbol ∘ denotes the Hadamard product, and ★ denotes circular convolution. Wr∈Rdr×d0 is regarded as a relation-specific matrix, learning interactions between the object node and its corresponding relations. dr is the total number of relations. Relation-aware attention mechanism: Referring to Equation (4), we define the MMKG as a graph with three types of edges: original, inverse, and self-loop. In order to represent each type of edge, we propose the direction-specific weights. Figure 8 depicts an example of a directed relational multi-modal knowledge subgraph. The solid arrows denote original relations, while dotted arrows denote inverse relations. The message passing function can be defined as follows: Wdir∈Rd1×d0 is a parameter specific to relation types that is defined by Equation (8). Here, O,I, and S represent original, inverse, and self-loop directions. d0 and d1 represent the input and output dimensions. To evaluate the varying importance of neighboring nodes during information aggregation, we adopt a single-layer dense projection to calculate the relation-aware attention of the adjacent neighboring set N(o) of the central node o: An example of a directed relational multi-modal knowledge subgraph. Each entity is equipped with three modalities and is connected by relational edges to form a directed graph. We parameterize the projection matrix by Watt∈R1×d1, while σ denotes the non-linear function LeakyReLU, attj represents the attention coefficient of the neighboring node j, which quantifies the importance of neighboring node j to the central node o under relation r. Subsequently, we use the Softmax function to normalize attj in order to stabilize the training process. The Softmax function transforms the attention coefficients of neighboring nodes into a probability distribution, ensuring that the sum of these weights is equal to 1: Then, the representation of the central node o is updated by calculating a weighted sum of the features from its neighboring nodes N(o) based on the normalized attention weights αj: To enhance the robustness and expressive power of DR-GAT, we apply the multi-head attention mechanism to capture the various features in multi-semantic spaces, assuming the proposed graph attention network has H attention heads and k stacked graph attention layers. By performing H independent attention calculations, each node will be equipped with H different node representations. To reduce the computational complexity, we average the outputs of the multi-head attention module rather than concatenating them to obtain the updated node representation eok: Finally, we define a learnable relational weight matrix Wr∈Rd1×d0 to update the relation embeddings in the k-th layer: Through the modal-specific directed relational graph attention networks, we complete the implicit missing modal features for all entities and obtain three modal-specific embeddings of entities and relations. 3.2.3. Multi-Stream Interaction Fusion After obtaining the structural, textual, visual, and the modal-specific relational embeddings of entities, we design an interaction fusion module to facilitate the information interaction among the modal features. Conventional methods typically employ concatenation or attention mechanism to combine multi-modal information. However, these methods are not capable enough to exploit the distinctive information or capture the complementary information among various modalities. Therefore, we introduce a low-rank tensor fusion mechanism [54] that effectively aids our approach to capture both intra-modal and inter-modal information. Firstly, we define the structural, textual, and visual features of the entity e as es,et,ev. Subsequently, an additional element 1 is added to each modal feature, which provides a learnable bias, enhancing the ability of the proposed framework to fit single-modal information. Then, a tensor cross product ⊗ is adopted to reconstruct tensor e into a high-dimensional tensor Z. The tensor cross product is utilized to fuse multi-modal information, where the first term represents the intra-modal of individual modal information. The second term captures the bi-modal interactions, and the last term indicates the tri-modal interactions for inter-modality fusion. Finally, the fused features can be obtained by: W denotes the weight matrix and b∈Rd1 denotes the bias. Nevertheless, this method requires the explicit creation of a high-dimensional tensor, which leads to exponential growth in tensor dimensions as the number of modalities increases, resulting in an exponential increase in space complexity. To address this issue, we adopt the low-rank decomposition multi-modal fusion strategy W¯ to substitute the weight tensor W to reduce the space complexity: k is a parameter utilized during the factorization, which is an artificially defined tensor rank. The factorization elements of the original weight tensor W are defined as wm(i); M refers to the number of modalities. m and i represent the m-th modal and the i-th factorization element. The fused embedding can be formed as: Λm=1M represents the element-wise multiplication across a sequence of tensors. In our model, M equals to 3, and therefore the formulation can be extended as Λm=1M=Λm=13em=es∘et∘ev. Equation (18) can be rewritten as: Following the above equations, we can efficiently compute the fused embeddings through the modal-specific factorization elements, notably reducing the computational complexity. Moreover, due to the scalability of the low-rank decomposition method, our multi-stream framework can flexibly extend to additional modalities (e.g., numerical or acoustic features). 3.2.4. Interpolation Re-Ranking Strategy Based on the modal-specific entity embeddings, relation embeddings, and the fused features encoded by the three-stream DR-GATs, we adopt a decoder to compute the predicted scores of factual triplets and utilize the proposed interpolation re-ranking strategy to integrate the decoding scores of the three modalities. The MMKGC task is finally completed by jointly optimizing the ensemble modal-specific binary cross-entropy logits. Definition of decoders: We propose two decoders to validate the effectiveness of MERGE according to ConvE [25] and InteractE [27]. ConvE is commonly used in GNN-based models for knowledge graph completion tasks and models the interactions between input entities and relations through a convolution operation. When presented with a triplet (s,r,o), ConvE transforms the embeddings of the subject entity s and the relation r into two-dimensional tensors, applying the convolution operation to compute the triplet score. The score function of ConvE for each factual triplet can be defined as: The subsequently proposed InteractE is an approach that enhances the expressive capability of ConvE through feature permutation, checkered feature reshaping, and circular convolution. The score function of InteractE for each factual triplet can be defined as: Here, ∗ refers to convolution operation, ⊛ represents depthwise circular convolution operation, w denotes convolutional kernel, and [;] denotes concatenation operation. ϕchk represents the reshaping operation used to rearrange the embedding es and er, so that no adjacent cells contain components from the same embedding. f and g represent the non-linear activation function ReLU and Sigmoid, respectively. The k-random permutations of both es and er are formed by Pk=[(es1,er1);…;(esk,erk)], which is designed to capture diverse and heterogeneous interactions between entities and relations. The scores of the factual triplets in each modality will be calculated by Equation (20) or Equation (21) with the single-modal entity embeddings esm, relation embeddings erm, and the fused entity embeddings eofused. The matrix W will be used to project the head-relation embeddings to the same dimensions as the fused embeddings. We conduct comparative experiments to evaluate the decoding performance of the two decoders in Section 4.5.4. Re-ranking strategy: Wang et al. [15] have observed that the best single-modal model often outperforms the joint model, which is contrary to common sense. Through a large number of experiments, they regard the original cause of such results to be overfitting. They provide solutions for this problem by optimizing the architecture of the model or employing pre-trained single-modal features to guide the training phase. Inspired by their theories, we propose a re-ranking strategy to ensemble the predicted scores achieved from each modal stream to decide the predicted entities. The ensemble of the modal ranks is not considered, as it may damage the modal-specific distribution characteristics. In fact, the distribution of the obtained scores could directly reflect the strengths and limitations of each modality for the final prediction. After sorting the scores in descending order, we regard the top-ranked entity as the final prediction result. The ensemble process of our framework is depicted in Figure 9, and the ensemble prediction score can be calculated as: ps, pt, and pv represent the prediction scores from structural, textual, and visual streams, respectively. The effect of the re-ranking strategy will be described in detail in Section 4.5.3. The hyper-parameters α∈[0,1], β∈[0,1], and γ∈[0,1] are proposed to control the trade-off among three objectives, which can be artificially adjusted. We consider the structural information as the essential information of the knowledge graph, therefore, setting α=1 to fully keep the structural predicted score. When β=γ=0, the model ignores the impact of textual and visual predicted results. β=γ=1 means an average weight is assigned to the predicted results from three modalities. By adjusting the parameters β and γ, we can adjust distinct weights to each of them. Section 4.5.5 exhibits the impact on different hyper-parameters β and γ. Interpolation re-ranking prediction with ensemble. e1,e2,…,en denote the indices of entities. ps,pt,pv denote the prediction scores of structural, textual, and visual modalities. α, β, and γ are proposed to control the trade-off among three objectives. Inference and learning: During the training phase, we utilize the binary cross-entropy with label smoothing as the loss function: ti represents the label, taking a value of 1 for the true factual triplet, while 0 to the opposite. m stands for the modality, N forms the total number of triplets, and pmi is the modal-specific corresponding scores to triplets. The linear combination of the logits in each modality is considered as the final training objective: 4. Experiment and Discussion 4.1. Datasets Our model is evaluated on four publicly available datasets: FB15k-237-IMG [55] (https://github.com/mniepert/mmkb, accessed on 11 May 2018), WN18RR-IMG [34] (https://github.com/wangmengsd/RSME, accessed on 14 January 2022), DB15K-IMG [55] (https://github.com/mniepert/mmkb, accessed on 11 May 2018), and MKG-W [38] (https://github.com/quqxui/MMRNS, accessed on 18 October 2022). The textual descriptions of the first two datasets are collected by [56], while the textual descriptions of the latter two datasets are collected from DBpedia [57]. Table 3 shows the details of these datasets: Table 3A summary of dataset statistics.DatasetEntityRelationTrainValidTestTextsImagesFB15k-237-IMG14,541237272,11517,53520,46614,515150,000WN18RR-IMG40,9431186,8353034313440,94370,385DB15K-IMG12,84227979,22299029904907812,818MKG-W15,00016934,1964276427414,46314,123 A summary of dataset statistics. FB15K-237-IMG: FB15K-237 [12] is a subset of the extensive knowledge graph Freebase [2] that excludes all inverse relations to address the influence of reversible relations. WN18RR-IMG: WN18 [4] is a subset originally extracted from WordNet [3], while WN18RR [25] is derived from the training set of WN18 by eliminating the inverse relations. DB15K-IMG: DB15K [55] is an open-source multi-modal knowledge graph, which is a subset of DBpedia [57]. MKG-W: MKG-W [38] is derived from Wikidata [1]. The images of entities are extended through web search engines. 4.2. Evaluation Metrics We employ three widely accepted metrics to evaluate the results of the MMKGC task. The following is a detailed description of these metrics. Here, N represents the total number of missing triplets, and ranki denotes the rank of the correct entity. For each test triplet, the model generates a ranked list of possible head/tail entities. The final result for each evaluation metric is obtained by averaging the head and tail prediction values: •Mean Reciprocal Rank (MRR): MRR calculates the average of the reciprocal ranks of the correct entities across all test samples. A higher MRR indicates a stronger ability of the model to rank the correct entity toward the top. MRR=1N∑iN1ranki Mean Reciprocal Rank (MRR): MRR calculates the average of the reciprocal ranks of the correct entities across all test samples. A higher MRR indicates a stronger ability of the model to rank the correct entity toward the top. MRR=1N∑iN1ranki •Mean Rank (MR): MR is the average rank of the correct entities across all test samples. A lower value indicates a better performance. MR=1N∑iNranki Mean Rank (MR): MR is the average rank of the correct entities across all test samples. A lower value indicates a better performance. MR=1N∑iNranki •Hits@k: Hits@k evaluates the proportion of correct answers appearing within the top-k candidates predicted by the model. A higher Hits@k value indicates a greater likelihood that the model includes the correct answer within its top-k predictions. In our method, k is set to 1, 3, and 10. Hits@k=∑iNranki