Lost in Embeddings: Information Loss in Vision–Language Models

Wenyan Li, Raphael Tang, Chengzu Li, Caiqi Zhang, Ivan Vulić, Anders Søgaard · 2025

Read Lost in Embeddings: Information Loss in Vision–Language Models by Wenyan Li, Raphael Tang, Chengzu Li, Caiqi Zhang, Ivan Vulić, Anders Søgaard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-11986.txt.

Introduction

Something is lost when a vision-language model squeezes an image into words. The picture first becomes vectors from a vision encoder. A connector then projects those vectors into the language model's space. Wenyan Li and five co-authors measure what survives that handoff in 2025.

One test tracks nearest neighbors among image codes before and after the projection step. The neighborhood reshuffles by about half. That loss tracks weaker retrieval scores. A second test rebuilds the first visual vectors from the projected ones, patch by patch. The result is a map of where the detail vanished. Those high-loss patches match the spots where the model later fails on visual questions. Builders get a clear warning sign, not just a bare score.

Catalog record

CREATORWenyan Li, Raphael Tang, Chengzu Li, Caiqi Zhang, Ivan Vulić, Anders Søgaard
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-11986

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lost in Embeddings: Information Loss in Vision–Language Models Wenyan Li, Raphael Tang, Chengzu Li, Caiqi Zhang, Ivan Vulić, Anders Søgaard Abstract Vision–language models (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the language model’s embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k k -nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k k -nearest neighbors diverging by 40–60% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle. 1 1 1 Code: https://github.com/lyan62/vlm-info-loss . 1 Introduction Vision–language models (VLMs) have advanced on many tasks, e.g., visual question answering and image captioning by combining pretrained vision encoders with pre-trained language models. Many of these models employ a small neural network module, known as a connector (or a projector), to bridge the gap between the visual and textual representation spaces. The connectors project visual representations into sequences of embeddings that a language model can process ( Chen et al. 2024a ; Liu et al. 2023 ; Deitke et al. 2024 ; Laurençon et al. 2024 ; Chen et al. 2024b ; Zhang et al. 2025 ; Sun et al. 2024 ) . Common connector architectures include multi-layer perceptrons (MLPs) or attention-based approaches Jaegle et al. 2021 ; Laurençon et al. 2024 . While these connector modules enable efficient cross-modal integration Li and Tang 2024 , projecting rich visual features into embeddings compatible with language models typically involves dimensional conversion and representation restructuring. Naturally, this raises questions about potential information loss 2 2 2 In this paper, we use “information loss” to broadly describe possible degradation of visual information, including aspects of the representation that cannot be recovered or directly observed after the projection. In a stricter sense, this could also be viewed as a representational gap or discrepancy rather than true loss, e.g. changes in the local geometry as reflected by k-nearest-neighbor relationships. during projection, and how such loss impacts downstream task performance. As shown in Figure 1 , the loss of critical visual details most relevant to answering the question imposes inherent limitations on the reasoning capabilities, since the language model’s performance is constrained by the quality and completeness of the visual information it receives. (a) Input image with red answer mask (b) Embedding norm signed difference (c) Image overlay with norm difference Figure 1: Visualization of patch-wise information loss in the embeddings explains the incorrect predicted answer in VizWiz Grounding VQA. For the question “What is the fifth number?”, LLaVA incorrectly predicted “18”. Figure 1(b) display the difference between the L^{2} norm of the original and the reconstructed patch embeddings. Blue regions indicate where original embeddings have larger norms than predicted embeddings, while red regions show where predicted embeddings have larger norms. The top 10 high-loss patches are marked by yellow squares. Figure 1(c) shows high loss occurring in several answer-relevant patches contribute to the incorrect prediction. Despite the growing research on VLM connector architectures and their impact on downstream performance Lin et al. 2024 ; Zhu et al. 2025 , there has been limited investigation into how well they preserve visual information in the latent space. Quantifying this information loss presents substantial challenges; traditional methods like canonical correlation analysis Hotelling 1936 struggle with variable-length high-dimensional visual features processed through diverse connector architectures in vision-language models. Performance degradation can also take more than one form, adding to the complexity of its study. For instance, it can take the form of a direct information loss due to an inherently lossy connector, or a geometric collapse where distinct features become entangled in the projected embedding space. To bridge this gap in the literature, we present an evaluation framework to quantify information loss in VLM connectors from both the geometric perspective and that of localized information loss. We first examine if the connector projection changes the geometric structure of latent visual representations. By introducing k -nearest neighbors overlap ratio , we measure how much the neighborhoods of image embeddings change before and after the projection in the latent representation space, thereby estimating how well geometric and semantic relationships are preserved. 3 3 3 Here, the semantic relationship denotes the semantic similarity between pairwise embeddings measured by L^{2} distance or their inner product similarity. Second, we quantify localized information loss by training a model to reconstruct the original visual embeddings from the projected embeddings. This patch-level visual embedding reconstruction allows us to pinpoint the high-loss regions in the image—areas where visual features are hard to recover after projection (Figure 1 ). This two-step approach provides both quantitative analysis and interpretable visualizations, offering insights into the nature of information transformation during vision-text integration. 2 VLMs and Connectors Integrating visual and textual inputs is fundamental for VLMs to process multimodal information effectively. Existing VLMs typically employ two main approaches Li and Tang 2024 : models like LLama3.2 gra 2024 and BLIP Li et al. 2023 leverage cross-modal attention mechanisms, while others such as LLaVA Liu et al. 2023 and Qwen-2-VL Bai et al. 2025 adopt connectors to project visual representations into latent vectors compatible with large language models (LLMs). 4 4 4 In this paper, we do not consider VQ-VAE van den Oord et al. 2017 based VLMs, which are more often used for text-to-image generation. Lin et al. 2024 categorize connectors into two types: feature-preserving and feature-compressing connectors. Feature-preserving connectors include MLPs that preserve the number of patch embeddings, such as the two-layer MLP connector in LLaVA. In contrast, feature compressing connectors project image patch embeddings to a shorter sequence, often involving transformer-based or convolution architectures with pooling operations over the original vision embedding ( Jaegle et al. 2021 ) . Feature compressing connectors include the perceiver sampler in Idefics2 Laurençon et al. 2024 and the patch merger in Qwen-2-VL Bai et al. 2025 . In this paper, we estimate information loss in both types of connectors. 2.1 Formalizing Encoders and Connectors We now give a formal definition of connectors. First, we consider the textual input. Let \Sigma be an alphabet of symbols. A string encoder , \phi , is a function that maps a string \sigma to a sequence of real-valued representations. Formally, \phi\colon\Sigma^{N}\rightarrow(\mathbb{R}^{D})^{N}, (1) where N\in\mathbb{N} is a parameter in the dependent type that denotes the length of the input string, and D is the dimensionality of the representation. Next, we turn to the visual input. Let \Delta=\{1,\ldots,256\}^{0pt\cdot 0pt\times C} be an array of image patches, where 0pt and 0pt represent the height and width dimensions, and C is the number of color channels per pixel. A two-dimensional image of patch dimensions M_{1}\times M_{2} can thus

Read the complete text