Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval
Guosheng Zhang, Linkai Liu, Keyao Wang, Haixiao Yue, Zhiwen Tan, Xiao TanBaidu Inc. {zhangguosheng, liulinkai, wangkeyao, yuehaixiao, tanzhiwen, tanxiao01}@baidu.com · 2026
Read Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval by Guosheng Zhang, Linkai Liu, Keyao Wang, Haixiao Yue, Zhiwen Tan, Xiao TanBaidu Inc. {zhangguosheng, liulinkai, wangkeyao, yuehaixiao, tanzhiwen, tanxiao01}@baidu.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25273.txt.
Introduction
Zhang and five colleagues propose Salient Subject-Aware Multimodal Embedding for unified retrieval, tackling visual neglect and semantic drift. Methods focus on sample objectives while missing subject semantics, so models fail to find salient regions and lean on text cues. Their framework uses multimodal models and visual experts to stress salient concepts, with a saliency goal aligning attention with key regions. A module adjusts features from saliency maps for balanced integration. Tests show state-of-the-art performance on MMEB.
Catalog record
| CREATOR | Guosheng Zhang, Linkai Liu, Keyao Wang, Haixiao Yue, Zhiwen Tan, Xiao TanBaidu Inc. {zhangguosheng, liulinkai, wangkeyao, yuehaixiao, tanzhiwen, tanxiao01}@baidu.com |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45320 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25273
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrieval Guosheng Zhang, Linkai Liu, Keyao Wang, Haixiao Yue, Zhiwen Tan, Xiao TanBaidu Inc. {zhangguosheng, liulinkai, wangkeyao, yuehaixiao, tanzhiwen, tanxiao01}@baidu.com Abstract Despite significant progress in Unified Multimodal Retrieval (UMR) powered by Large Multimodal Models (LMMs), existing embedding methods primarily focus on sample-level objectives via contrastive learning while overlooking the the crucial subject-level semantics. This limitation hinders the model’s ability to group semantically coherent subjects in complex multimodal queries, manifesting as semantic alignment deviation —where models fail to accurately localize salient text-referred regions in visual content. Moreover, without explicit guidance to model salient visual subjects, LMMs tend to over-rely on textual cues, resulting in visual modality neglect and suboptimal utilization of visual knowledge. To this end, we propose S alient S ubject- A ware M ultimodal E mbedding (SSA-ME), a novel framework designed to enhance fine-grained representation learning through saliency-aware modeling. SSA-ME leverages LMMs and visual experts to identify and emphasize salient visual concepts in image-text pairs, and introduces a saliency-guided objective to better align cross-modal attention with semantically meaningful regions. Additionally, a feature regeneration module recalibrates visual features based on the derived saliency maps, ensuring a balanced and semantically coherent integration across modalities. Extensive experiments show that our method achieves state-of-the-art performance on the MMEB benchmark, demonstrateting that incorporating subject-level modeling substantially improves multimodal retrieval. Comprehensive qualitative analyses further illustrate the interpretability and effectiveness of our approach. Figure 1 : The figure demonstrates two key limitations in multimodal embedding models. In (a), we observe significant semantic alignment deviation in the baseline model, which fails to accurately localize text-referred subjects. In contrast, our SSA-ME method successfully focuses on the semantically relevant visual subjects. The scatter plot in (b) reveals the visual modality neglect problem, where baseline models cluster (blue points) indicating excessive textual attention at the expense of visual information. Our method achieves a more balanced attention distribution between visual and textual modalities, as evidenced by the centered cluster distribution. † † footnotetext: ∗ Equal contribution. † † footnotetext: Work done during Linkai Liu’s internship at Baidu Inc. 1 Introduction Universal Multimodal Retrieval (UMR) 37 ; 15 ; 23 ; 14 ; 11 ; 28 ; 19 ; 1 has emerged as a critical paradigm for cross-modal information systems, enabling unified representation learning across diverse modalities through Large Multimodal Models (LMMs). While existing methods have made significant progress by leveraging contrastive learning at the sample level, their predominant focus on instance discrimination overlooks the essential need for subject-level semantic understanding.This fundamental limitation becomes particularly apparent when handling complex multimodal queries that require fine-grained correspondence between visual subjects and textual references. As illustrated in Figure 1 (a), we observe significant semantic alignment deviation in existing methods. When processing queries involving different textual subjects, baseline models fail to establish precise visual-textual correspondences, often misallocating attention to semantically irrelevant regions rather than focusing on the actual subjects mentioned in the text. Moreover, the lack of explicit mechanisms to guide models toward salient visual subjects leads to a systematic over-reliance on textual cues—a phenomenon we term visual modality neglect 20 ; 7 . As shown in Figure 1 (b), baseline embeddings cluster predominantly in text-biased regions, indicating that multimodal representations are constructed largely from textual information while visual content remains underutilized. This imbalance fundamentally limits the model’s capacity to leverage complementary visual information effectively. These interconnected limitations—semantic misalignment due to insufficient subject-level modeling and modality imbalance arising from the lack of explicit visual subject guidance—motivate our investigation into a more fine-grained approach that explicitly incorporates subject-aware reasoning into multimodal representation learning. In this paper, we propose Salient Subject-Aware Multimodal Embedding (SSA-ME), a novel framework that enhances multimodal representation learning at the subject level through saliency-aware mechanisms. First, we introduce a Saliency-Guided Attention Alignment (SGA) mechanism that leverages the powerful multimodal understanding capability of LMMs combined with visual expert models’ object perception to precisely localize salient subjects in image-text pairs. A saliency-guided loss function is designed to distill the saliency detection capability into our model through latent attention alignment, thereby significantly improving the semantic correlation between visual and textual modalities. Furthermore, we develop a Saliency-Driven Feature Regeneration (SDR) mechanism that selectively recycles visual features based on the model’s predicted saliency maps and fuses them with the global multimodal embedding. This approach fully exploits salient core features in the visual modality, enhancing the model’s attention weight on visual inputs. As demonstrated in Fig. 1 (a), our method accurately localizes text-mentioned salient subjects in images. Moreover, Fig. 1 (b) shows that the attention distribution shifts toward the central balance line, indicating improved equilibrium between visual and textual modalities. In summary, our contributions are threefold: (1) We introduce a subject-level perspective for analyzing multimodal embedding learning, through which we identify and systematically examine two critical yet understudied limitations: semantic alignment deviation and visual modality neglect. (2) We propose two simple yet effective components: SGA and SDR. These components enable the model to better focus on salient visual regions and mitigate visual modality neglect. (3) Our approach achieves state-of-the-art performance on the MMEB benchmark. Experimental results confirm that explicitly modeling salient subjects significantly improves retrieval performance, demonstrating the practical value of our subject-level perspective. Figure 2 : Overview of Salient Subject-Aware Multimodal Embedding (SSA-ME) framework. Black and blue arrows represent the training flow, while only blue arrows indicate the inference flow. To ensure the extracted salient subjects are semantically representative, we feed the query pair along with its corresponding positive sample into Qwen2.5-VL-7B 2 , leveraging a carefully designed prompt to guarantee that the identified salient subjects align meaningfully with the content of the positive sample. 2 Related Works Universal Multimodal Retrieval. Early approaches to multimodal retrieval primarily focused on unimodal settings, whereas contemporary multimedia applications increasingly demand systems capable of processing hybrid image-text inputs and generating corresponding multimodal responses. A common pipeline involves leveraging pre-trained vision-language models such as CLIP 26 and BLIP 22 for feature extraction, combined with fusion modules as seen in UniVL-DR 24 and UniIR 32 . Despite their effectiveness, these methods struggle in complex retrieval scenarios such as composed image retrieval 25 ; 33 ; 3 ; 29 , long-text-to-image retrieval 35 , and image-question-to-multimodal document retrieval 5 ; 12 . The field has since shifted toward leveraging Large Multimodal Models (LMMs) for their superior cross-modal understanding. Methods such as E5-V 14 employ carefully designed prompts to project images and text into a unified latent space, achieving strong zero-shot retrieval via text-pair fine-tuning 9 . GME 37 pioneered unified retrieval for visual documents and introduced a data synthesis pipeline for generating large-scale multimodal training data. More recently, VLM2Vec 15 advanced the state-of-the-art on the MMEB benchmark by fine-tuning Qwen2-VL 30 with Explicit One-token Limitation (EOL), while LLaVE 19 enhanced embedding discrimination for hard negative samples. Nevertheless, these approaches remain constrained by sample-level optimization through positive-negative contrastive learning, leaving the potential of subject-level multimodal embedding largely unexplored. Fine-Grained Visual Understanding in LMM Enabling LMMs to achieve fine-grained visual understanding has emerged as a critical research direction. Recent studies highlight persistent limitations in their ability to establish detailed visual-textual correspondences 17 ; 16 ; 8 ; 20 ; 7 . Kang et al. 17