Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models

Jiachen Jiang, Jinxin Zhou, Bo Peng, Xia Ning, Zhihui Zhu · 2025

Read Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models by Jiachen Jiang, Jinxin Zhou, Bo Peng, Xia Ning, Zhihui Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17316.txt.

Introduction

Jiachen Jiang and co-authors probe how multimodal language models actually read pictures. A small projector layer links a vision encoder to the language model, but what it passes along was unclear. The team shows the projector squeezes visual data hard, dropping redundant detail while keeping what the model needs to grasp a scene.

They then test patch-level alignment, the match between each image patch and the words for what it shows. Training on captions improves that match only weakly, leaving it coarse. Their fix, patch-aligned training, sharpens the match at low cost. Models trained this way wrote better captions and gained on grounding, question answering, and other tasks. The work gives builders a clearer view of the vision-language bridge.

Catalog record

CREATORJiachen Jiang, Jinxin Zhou, Bo Peng, Xia Ning, Zhihui Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17316

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models Jiachen Jiang, Jinxin Zhou, Bo Peng, Xia Ning, Zhihui Zhu Abstract Achieving better alignment between vision embeddings and Large Language Models (LLMs) is crucial for enhancing the abilities of Multimodal LLMs (MLLMs), particularly for recent models that rely on powerful pretrained vision encoders and LLMs. A common approach to connect the pretrained vision encoder and LLM is through a projector applied after the vision encoder. However, the projector is often trained to enable the LLM to generate captions, and hence the mechanism by which LLMs understand each vision token remains unclear. In this work, we first investigate the role of the projector in compressing vision embeddings and aligning them with word embeddings. We show that the projector significantly compresses visual information, removing redundant details while preserving essential elements necessary for the LLM to understand visual content. We then examine patch-level alignment—the alignment between each vision patch and its corresponding semantic words—and propose a multi-semantic alignment hypothesis . Our analysis indicates that the projector trained by caption loss improves patch-level alignment but only to a limited extent, resulting in weak and coarse alignment. To address this issue, we propose patch-aligned training to efficiently enhance patch-level alignment. Our experiments show that patch-aligned training (1) achieves stronger compression capability and improved patch-level alignment, enabling the MLLM to generate higher-quality captions, (2) improves the MLLM’s performance by 16% on referring expression grounding tasks, 4% on question-answering tasks, and 3% on modern instruction-following benchmarks when using the same supervised fine-tuning (SFT) setting. The proposed method can be easily extended to other multimodal models. 1 Introduction Multimodal Large Language Models (MLLMs) [ 1 , 2 , 3 , 4 , 5 , 6 , 7 ] have recently gained significant attention and made notable progress. These models possess the ability to process and understand both visual and textual information, enabling them to perform complex reasoning [ 8 , 9 ] , generate textual descriptions from images [ 10 ] , and answer image-related questions [ 11 ] . Consider an MLLM \mathcal{M}=(\mathcal{E},\mathcal{L},\mathcal{P}) where \mathcal{E} is the vision encoder, \mathcal{L} is the LLM, and \mathcal{P} is the lightweight projector that connects the two parts. The standardized training paradigm follows two key phases: pretraining and instruction-tuning [ 1 , 2 ] . During the pretraining phase, only the lightweight projector \mathcal{P} is trained leaving the vision encoder \mathcal{E} and the LLM \mathcal{L} frozen. In instruction-tuning stage, both projector \mathcal{P} and the LLM \mathcal{L} are trainable. Despite the remarkable progress achieved through this training paradigm, recent works [ 12 , 13 , 14 , 15 , 16 ] reveal that these MLLMs still struggle with region-specific understanding and tend to hallucinate with irrelevant or incorrect information. The reason of these issues remains an active area. In addition to limitations in the vision encoder or the capabilities of the language model, a significant contributing factor lies in the projector [ 4 , 17 ] . Figure 1 : The patch-level alignment is measured in two aspects: Left) patch-level localization . Using LLM word embeddings of labels L_{\text{GT}} , we predict the most relevant image regions by calculating cosine similarity with the vision embedding. Right) multi-semantic alignment . Using matching pursuit, we decompose each vision embedding into several discrete words by treating LLM word embeddings as a basis. Since the vision embedding are obtained after the MLLM projector, we compare across three projectors: random projector \mathcal{P}_{\text{Random}} (top), LLaVA1.5 projector \mathcal{P}_{\text{LLaVA}} (middle), and our patch-aligned projector \mathcal{P}_{\text{Patch Aligned}} (bottom). Results show that \mathcal{P}_{\text{LLaVA}} exhibits weak patch-level alignment abilities, while our \mathcal{P}_{\text{Patch Aligned}} significantly enhances these two aspects. The platonic representation hypothesis [ 18 ] suggests that representations in deep networks are converging across data modalities 1 1 1 In the sense that vision and language models measure the distance between data points in a similar way. , implying a shared structural alignment. However, for a vision encoder \mathcal{E} and an LLM \mathcal{L} that are pretrained separately, there is no guarantee that the embedding space induced by \mathcal{E} will share the same basis as the embedding matrix \bm{W} in \mathcal{L} , even if they have the same dimensionality. Thus, as the only connection between the two modalities, the projector \mathcal{P} plays a crucial role. However, current work remains at a superficial understanding that the projector performs alignment, lacking a thorough and systematic analysis of its function. Thus, in this paper, we are motivated by the following questions: How does the projector align multimodal information in MLLMs? How to quantify and improve alignment in current models? Contributions. In this work, we provide a detailed analysis of the alignment between each vision patch and its corresponding semantic words and develop methods to improve patch-level alignment, enabling the LLM to better understand visual content in the input space. Our contributions can be summarized as follows, Projector compresses visual information. Vision embeddings are naturally continuous and contain redundant information, whereas LLM input word embeddings are discrete. Thus, a natural question arises: Is the information contained in the vision embeddings compressed through the projector? To address this question, we propose quantifying the amount of information contained in the embeddings using Von Neumann entropy [ 19 ] . Our experiments show that information is significantly compressed after projection, indicating that the projector plays a crucial role in eliminating redundant information while preserving the essential elements needed for the LLM to understand the visual content. Analyzing patch-level alignment. We then examine each image patch in detail and study how its embedding aligns with the corresponding text embedding. However, challenges arise due to (1) a lack of text labels for each image patch and (2) the possibility that each image patch contains multiple semantic meanings. To address these challenges, we propose two complementary approaches to quantitatively and qualitatively study patch-level alignment. We first focus on alignment with respect to the objects in the image. Specifically, for an input image \bm{X} with embedding \bm{V}=\mathcal{P}\circ\mathcal{E}(\bm{X}) , which serves as input to an LLM \mathcal{L} , we propose a patch-level alignment measure \operatorname{Align}(\bm{V},\bm{W}) to quantify the alignment between \bm{V} and the word embedding \bm{W} . A higher \operatorname{Align}(\bm{V},\bm{W}) indicates a greater ability of the LLM to identify objects, even in the word embedding space. Inspired by previous work on decomposing word embedding vectors [ 20 , 21 ] , we then propose a multi-semantic alignment hypothesis, which states that the embedding for each vision patch can be decomposed as a linear combination of word embeddings corresponding to all semantic meanings within the patch. To verify this hypothesis, we apply the matching pursuit algorithm [ 22 ] to identify the most relevant tokens from the LLM dictionary for each vision patch. As shown in Figure 1 , our analysis indicates that the LLaVA projector improves patch-level alignment but only to a limited extent, resulting in weak and coarse alignment. This is due to (1) the caption loss only implicitly enforces token-level alignment, (2) captions tend to be short and primarily focus on a few prominent regions of interest, often neglecting many other regions. Consequently, numerous visual tokens (e.g., floor and TV cabinet) are often aligned with meaningless or garbled words. Patch-Aligned Training for Improving

Read the complete text