Multi-level Matching Network for Multimodal Entity Linking
Zhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. Pan · 2024
Read Multi-level Matching Network for Multimodal Entity Linking by Zhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. Pan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10440.txt.
Introduction
Zhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, and Jeff Pan wrote a 2024 paper on linking names in text and images to the right entries in a knowledge base. A photo caption may mention a place or person that words alone cannot pin down. Pictures help, but the two kinds of clues must be matched with care.
Their network matches at several levels. It first builds strong features for text and image apart, using contrast within each kind to sharpen them. It then matches across kinds at both broad and fine levels. Older methods missed negative examples within one kind and let information flow only one way. The design aims to fix both gaps, which should cut wrong links in news and social media tools.
Catalog record
| CREATOR | Zhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. Pan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10440
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-level Matching Network for Multimodal Entity Linking Zhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. Pan Abstract . Multimodal entity linking (MEL) aims to link ambiguous mentions within multimodal contexts to corresponding entities in a multimodal knowledge base. Most existing approaches to MEL are based on representation learning or vision-and-language pre-training mechanisms for exploring the complementary effect among multiple modalities. However, these methods suffer from two limitations. On the one hand, they overlook the possibility of considering negative samples from the same modality. On the other hand, they lack mechanisms to capture bidirectional cross-modal interaction. To address these issues, we propose a M ulti-level M atching network for M ultimodal E ntity L inking ( M 3 EL {\rm\textbf{M}^{\textbf{3}}\textbf{EL}} ). Specifically, M 3 EL {\rm M^{3}EL} is composed of three different modules: ( i ) a Multimodal Feature Extraction module, which extracts modality-specific representations with a multimodal encoder and introduces an intra-modal contrastive learning sub-module to obtain better discriminative embeddings based on uni-modal differences; ( ii ) an Intra-modal Matching Network module, which contains two levels of matching granularity: Coarse-grained Global-to-Global and Fine-grained Global-to-Local , to achieve local and global level intra-modal interaction; ( iii ) a Cross-modal Matching Network module, which applies bidirectional strategies, Textual-to-Visual and Visual-to-Textual matching, to implement bidirectional cross-modal interaction. Extensive experiments conducted on WikiMEL, RichpediaMEL, and WikiDiverse datasets demonstrate the outstanding performance of M 3 EL {\rm M^{3}EL} when compared to the state-of-the-art baselines. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation emai; August 03–07, 2025; Toronto, Canada Woodstock ’18: ACM Symposium on Neural Gaze Detection, August 03–07, 2025, Toronto, Canada ISBN: 978-1-4503-XXXX-X/18/06 CCS: Information systems Multimedia databases CCS: Information systems Multimedia information systems CCS: Information systems Data mining CCS: Computing methodologies Knowledge representation and reasoning 1. Introduction Figure 1 . An example of MEL. Dotted boxes of different colors represent different features: color purple for mention textual description (mention text), color orange for mention visual context (mention image), color green for entity textual description (entity text), color blue for entity visual context (entity image). Entity Linking (EL) aims at aligning the mentions within a context to the corresponding entities in a knowledge base ( Hoffart et al., 2011 ; Shen et al., 2015 ) . EL supports numerous downstream information-retrieval applications, such as question answering ( Yih et al., 2015 ; Xiong et al., 2019 ; Longpre et al., 2021 ; Hu et al., 2022 ; Yan et al., 2024 ) , semantic search ( Cheng et al., 2007 ; Bordino et al., 2013 ; He et al., 2023 ) , dialog systems ( Liao et al., 2018 ; Ahmadvand et al., 2019 ) , and so on ( Hu et al., 2024 ; Han et al., 2024 ) . Most existing EL frameworks focus on mention disambiguation via context resolution in the textual modality. However, for multimodal information including images along with text, conventional text-based approaches to EL struggle to effectively encode such complex content. Multimodal Entity Linking (MEL) extends traditional EL by considering multimodal information, i.e., it aims at linking textual and visual mentions into their corresponding entities in a multimodal knowledge base. For example, in Figure 1 , the mention Black Widow can be linked to the entities Animal_Black_Widow , Movie_Black_Widow and Song_Black_Widow . Although, it is possible to roughly guess the entity to which the mention has to be linked by relying solely on the textual description, the associated visual information can be used to improve the confidence of text-based predictions. Indeed, the image associated to the mention Black Widow has a high degree of semantic similarity with that of the entity Movie_Black_Widow , both including the actress Scarlett Johansson , which substantially increases the probability that the mention Black Widow is linked to the entity Movie_Black_Widow . A wide variety of approaches to MEL have been already proposed, including representation learning (RL) frameworks ( Moon et al., 2018 ; Adjali et al., 2020 ; Zheng et al., 2022 ; Wang et al., 2022b ; Xing et al., 2023 ; Yang et al., 2023 ) , vision-and-language pre-training (VLP) based methods ( Radford et al., 2021 ; Kim et al., 2021 ; Li et al., 2021 ; Dou et al., 2022 ; Luo et al., 2023 ) , and a generative-based method ( Shi et al., 2024 ) . Despite the substantial progress achieved so far, each of these approaches have their own disadvantages. The generative-based method GEMEL ( Shi et al., 2024 ) leverages the in-context learning capability of large language models (LLMs) to create demonstrations, which requires a substantial amount of time and space overhead, and its performance is unsatisfactory. RL-based methods explore complementary effects of different modalities via joint or collaborative representation learning, e.g., using the concatenation operation ( Adjali et al., 2020 ) , additive attention ( Moon et al., 2018 ) or cross-attention mechanisms ( Wang et al., 2022b ) . However, the above methods fail to integrate the information from pre-trained visual and textual models into the model representation process. VLP-based methods use a model jointly pre-trained on visual and language tasks as a modal encoder, although it can better understand the association between textual and visual information, they still suffer of two problems: (1) Diversity of Negative Samples. To learn better textual and visual embeddings, VLP models like CLIP ( Radford et al., 2021 ) introduce a contrastive loss bringing paired image-text representations together, while pushing unpaired instances away from each other. However, they overlook the possibility of using negative samples from the same modality. Indeed, VLP-based MEL methods directly apply a VLP model as the textual and visual encoder, disregarding intra-modal negative samples. However, for the MEL task, both mentions and candidate entities contain independent textual and visual information. Implementing entity-mention interaction within the textual or visual modality is a natural way to improve the model representation capabilities. (2) Bidirectional Cross-modal Interaction. Existing methods only consider a one-way information flow, from textual to visual or visual to textual, during cross-modal interaction, i.e., they lack bidirectional cross-modal interaction. In practice, the textual-to-visual flow focuses more on the impact of the textual knowledge on the visual modality, while visual-to-textual pays attention to applying visual knowledge to the text modality. Therefore, we advocate that it is necessary to design a mechanism that better captures the bidirectional interaction pattern, to fully harness the knowledge of all available modalities. To address the above two shortcomings, we propose a M ulti-level M atching network for M ultimodal E ntity L inking ( {\rm\textbf{M}^{\textbf{3}}\textbf{EL}} ). {\rm M^{3}EL} includes the following three modules. The Multimodal Feature Extraction module utilizes a pre-trained CLIP model to obtain modality-specific representations for each entity and mention. To alleviate shortcoming (1), we introduce an Intra-modal Contrastive Learning module to include negative samples within a modality into the discriminative embedding acquisition process. Additionally, we employ an Intra-modal Matching Network module which contains two levels of matching granularity, Coarse-grained Global-to-Global and Fine-grained Global-to-Local matching, to realize the interaction between local and global features within a modality. Moreover, we design a Cross-modal Matching Network module which considers bidirectional matching strategies, Textual-to-Visual and Visual-to-Textual matching, to reduce the gap