Multi-level Mixture of Experts for Multimodal Entity Linking Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25),

Zhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li, Jeff Z. Pan · 2025

Read Multi-level Mixture of Experts for Multimodal Entity Linking Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25), by Zhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li, Jeff Z. Pan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07108.txt.

Introduction

Zhiwei Hu and co-authors link names in mixed text-and-image data to the right entries in a knowledge base. Their 2025 paper tackles multimodal entity linking, where a short mention may be vague and its image may matter more or less by case. Older systems fuse the two streams in a fixed way.

Their model routes features through mixtures of expert subnetworks at two levels. A first module fattens thin mentions with matching encyclopedia descriptions found by a language model. Experts then pick the useful parts of each stream on the fly. Tests show clear gains over prior methods. The authors credit the dynamic choice of what to trust.

Catalog record

CREATORZhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li, Jeff Z. Pan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-07108

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-level Mixture of Experts for Multimodal Entity Linking Conference: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 3–7, 2025; Toronto, ON, Canada Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’25), August 3–7, 2025, Toronto, ON, Canada DOI: 10.1145/3711896.3737060 ISBN: 979-8-4007-1454-2/2025/08 Zhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li, Jeff Z. Pan Abstract . Multimodal Entity Linking (MEL) aims to link ambiguous mentions within multimodal contexts to associated entities in a multimodal knowledge base. Existing approaches to MEL introduce multimodal interaction and fusion mechanisms to bridge the modality gap and enable multi-grained semantic matching. However, they do not address two important problems: ( i i ) mention ambiguity , i.e., the lack of semantic content caused by the brevity and omission of key information in the mention’s textual context; ( i ​ i ii ) dynamic selection of modal content , i.e., to dynamically distinguish the importance of different parts of modal information. To mitigate these issues, we propose a M ulti-level M ixture o f E xperts ( MMoE ) model for MEL. MMoE has four components: ( i i ) the description-aware mention enhancement module leverages large language models to identify the WikiData descriptions that best match a mention, considering the mention’s textual context; ( i ​ i ii ) the multimodal feature extraction module adopts multimodal feature encoders to obtain textual and visual embeddings for both mentions and entities; ( i ​ i ​ i iii )-( i ​ v iv ) the intra-level mixture of experts and inter-level mixture of experts modules apply a switch mixture of experts mechanism to dynamically and adaptively select features from relevant regions of information. Extensive experiments demonstrate the outstanding performance of MMoE compared to the state-of-the-art. MMoE’s code is available at: https://github.com/zhiweihu1103/MEL-MMoE . † † cc-license: by 1. Introduction Figure 1 . (a) The given mention Black Panther within a multimodal context. For mention textual context m_{t} , different color depths indicate the importance for the prediction of the mention m_{w} . (b) Wikidata descriptions of entities called Black Panther . The number of \bigstar in m_{v} indicates the importance level of the corresponding region. Entity linking (EL) ( Ayoola et al. 2022 ) is the task of recognising mentions of entities in unstructured textual content and linking them to the corresponding entities in a knowledge graph (KG) ( Pan et al. 2017b ; Pan et al. 2017a ) . EL supports various downstream information retrieval tasks, such as question answering ( Xiong et al. 2019 ; Fawei et al. 2020 ; Longpre et al. 2021 ; Huang et al. 2025 ) , semantic search ( Thomas et al. 2007 ; Pan et al. 2009 ; Bordino et al. 2013 ; He et al. 2023 ; Huang et al. 2025 ) , dialog systems ( Liao et al. 2018 ; Ahmadvand et al. 2019 ; Wang et al. 2025 ) , and so on ( Chen et al. 2018 ; Pan et al. 2019 ; Hu et al. 2024b ) . With the increasing availability of multimodal content, multimodal entity linking (MEL), which aims to link ambiguous mentions in multimodal contexts to entities in a multimodal KG (MMKG), has garnered increasing attention. Figure 1 shows that to successfully link the mention Black Panther to its corresponding entities { m_{q}^{1},m_{q}^{2},m_{q}^{3} }, a comprehensive understanding of the associated textual m_{t} and visual m_{v} information is required. Several approaches have been introduced to handle the MEL task, including ( i ): vision-and-language pre-trained ( Radford et al. 2021 ; Kim et al. 2021 ; Li et al. 2021 ; Dou et al. 2022 ) , ( ii ): generative-based ( OpenAI 2023 ; Shi et al. 2024 ; Long et al. 2024 ) , and ( iii ): multimodal interaction-based ( Moon et al. 2018 ; Adjali et al. 2020 ; Zheng et al. 2022 ; Wang et al. 2022b ; Luo et al. 2023 ; Luo et al. 2024 ; Hu et al. 2024a ; Sui et al. 2024 ; Zhang et al. 2024 ) . These frameworks have made steady progress, obtaining performance improvements in various benchmarks like WikiMEL ( Wang et al. 2022b ) , RichpediaMEL ( Wang et al. 2022b ) , and WikiDiverse ( Wang et al. 2022a ) . However, each of these approaches have their own disadvantages: vision-and-language pre-trained based methods often neglect the fine-grained interactions within modality-specific information, resulting in an inability to fully uncover the latent relationships among the internal elements of a modality. Generative-based methods face challenges in associating visual inputs with their internal entity knowledge, making it difficult to establish connections between fine-grained visual patterns and the corresponding entities. Although the prevailing approaches are based on multimodal interaction, they still suffer from two crucial issues: \triangleright Mention Ambiguity. The context in which textual mentions occur is frequently concise and may incorporate abbreviations, idiomatic expressions, or domain-specific jargon. This scarcity and ambiguity of contextual information can render the accurate identification of the referenced entity infeasible, directly impacting the performance of a model. Figure 1 shows an intuitive example in which for the mention Black Panther , it is difficult to determine whether Black Panther is an animal , a movie or a band simply based on its given textual context m_{t} . Indeed, m_{t} might relate to the film’s excellence or the actors’ outstanding performance, the animal’s striking movements, or the band’s impressive live performance. \triangleright Dynamic Selection of Modal Content. Existing methods typically encode the full textual sequence or visual image into a single fixed-length vector for modal interaction, ignoring that different sections within a modality contribute differently to the mention prediction of the corresponding entity. Figure 1 presents a typical example illustrating the variation in the significance of different regions within the image and sections of the textual sequence. For instance, for the textual context m_{t} , compared to the definite article the , the phrase unforgettable performance exerts a greater impact on the interpretation of the mention Black Panther . Similarly, for the visual modality knowledge m_{v} , the regions corresponding to actor Chadwick Aaron Boseman and Marvel Studios should receive more attention compared to the region of actor Danai Jekesai Gurira . Thus, it is paramount to dynamically distinguish the importance of different parts of modal information. To address these two problems, we propose a M ulti-level M ixture o f E xperts ( MMoE ) model for the MEL task, composed of the following four components. The description-aware mention enhancement module leverages large language models to identify the WikiData descriptions that best match a mention, to compensate for the limitation of concise mentions. Then, the multimodal feature extraction module utilizes a pre-trained CLIP model to represent textual tokens and visual patches for both mentions and entities. Afterwards, we design a switch mixture of experts (SMoE) mechanism to dynamically and adaptively select features from relevant regions of information. Specifically, we introduce SMoE for both intra and inter modality interaction. In summary, our main contributions are three-fold: • We propose the MMoE framework for MEL which selects the optimal expert network from the intra- and inter-modality

Read the complete text