Positional Bias in Multimodal Embedding Models: Do They Favor the Beginning, the Middle, or the End?

Kebin Wu, Fatima Albreiki · 2025

Read Positional Bias in Multimodal Embedding Models: Do They Favor the Beginning, the Middle, or the End? by Kebin Wu, Fatima Albreiki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11216.txt.

Introduction

Kebin Wu and Fatima Albreiki investigate positional bias in multimodal embedding models, the tendency to favor content at certain spots in an input regardless of what it says. Such bias is well studied in text generators but barely explored in representation models, and even less in models that join images and text. The authors study the bias in image-text retrieval tasks.

They first separate true content importance from pure position effects, then measure bias across models and data sets. Bias proved widespread but split by modality: text encoders favored the start, while image encoders favored both the start and the end. Encoding scheme, training loss, and paired training all feed the effect.

Catalog record

CREATORKebin Wu, Fatima Albreiki
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11216

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Positional Bias in Multimodal Embedding Models: Do They Favor the Beginning, the Middle, or the End? Kebin Wu, Fatima Albreiki Abstract Positional bias—where models overemphasize certain positions regardless of content—has been shown to negatively impact model performance across various tasks. While recent research has extensively examined positional bias in text generation models, its presence and effects in representation models remain underexplored. Even less is known about such biases in multimodal models. In this work, we investigate positional bias in multimodal representation models, specifically in the context of image-text retrieval. We begin by distinguishing between context importance and positional bias, and then assess the presence and extent of positional bias across different models and datasets. Our experiments demonstrate that positional bias is prevalent in multimodal models, but manifests differently across modalities: text encoders tend to exhibit bias toward the beginning of the input, whereas image encoders show bias at both the beginning and end. Furthermore, we find that this bias arises from, or is amplified by, a combination of factors, including the positional encoding scheme, training loss, context importance, and the nature of using image-text pairs in multimodal training. Code — https://github.com/tiiuae/PosBias/ 1 Introduction Transformer-based models have achieved notable success, especially in natural language processing, leading to the development of numerous language models. However, emerging research indicates that their ability to capture contextual information is influenced by the position of that information within the input sequence—an issue known as positional bias. For example, Liu et al. ( 13 ) showed that models often prioritize content at the beginning or end, while neglecting the middle—a phenomenon termed “lost in the middle.” This weakens their reasoning abilities and introduces instability in evaluations. In response, many studies have explored the underlying causes of positional bias and proposed mitigation strategies. Despite recent progress, research on positional bias has largely focused on text generation. We extend this by analyzing positional bias in multimodal representation learning, specifically within cross-modal retrieval tasks. Our study examines representative multimodal embedding models such as CLIP ( 17 ) and evaluates positional bias in their text and image encoders separately. Figure 1 shows a text-to-image retrieval example where shifting the position of the ground-truth text within the query significantly alters the retrieved results. We begin by distinguishing between positional bias and contextual importance. To investigate positional bias, we segment the image or text into multiple parts and iteratively shift a selected segment across different positions, while replacing the remaining segments with either dummy perturbations or modality- and model-specific masking content. This approach reveals substantial positional biases in both the image and text encoders of multimodal models. We extend the analysis to several multimodal models and investigate the prevalence and causes of such bias in multimodal representation learning. In this paper, we make the following contributions. First, we reveal the presence of positional bias in multimodal models—a phenomenon not previously reported, to our knowledge. Second, we empirically characterize distinct bias patterns: in text encoders, performance is consistently higher when a segment appears at the beginning of the sequence; in image encoders, bias is more variable, favoring either the beginning or both ends. Third, through extensive experiments, we show that this bias persists across a wide range of models, regardless of data distribution, positional encoding, training length, model size, resolution, patch size, loss function, or vision encoder architecture. Finally, we identify several contributing factors that may drive or modulate positional bias. Figure 1: Top-5 text-to-image retrieval results. The caption is divided into six segments. Each row corresponds to a different position where only Segment 0 (highlighted in dark green) is placed, while the remaining positions are masked. Across the six rows, Segment 0 is shifted through all six possible positions. 2 Related Work 2.1 Positional Bias in Text Generation Positional bias has been extensively studied in text generation models, especially in the context of long-context language modeling. Liu et al. ( 13 ) identified the “lost in the middle” phenomenon, where language models underperform when relevant information appears mid-sequence. Hsieh et al. ( 8 ) found that large language models often exhibit a U-shaped attention pattern, emphasizing the beginning and end of sequences over the middle, regardless of semantic importance. Several mitigation strategies have been proposed, including attention calibration ( 8 ) , data augmentation ( 7 ) , differential attention mechanisms ( 24 ) , and replacing causal attention with bidirectional variants ( 21 ) . While some studies have extended this analysis to vision-language models ( 19 ) , their focus remains on text generation tasks like image captioning. 2.2 Positional Bias in Text Representation Positional bias has been observed not only in generation tasks but also in text representation models used for retrieval and classification. Coelho et al. ( 3 ) investigated this bias in web document retrieval and found that such models tend to prioritize the beginning of documents. They attribute this effect to contrastive training and the inverted pyramid structure commonly found in writing. Similar head-biased tendencies were found in Named Entity Recognition (NER), Part-of-Speech (POS) tagging ( 1 ) , and extractive question answering ( 9 ) . Goel et al. ( 6 ) analyzed this bias through semantic similarity measurement, showing consistent overemphasis on early tokens across diverse embedding models. These biases have been hypothesized to result from characteristics of the training data and preprocessing steps such as input truncation. In this paper, we extend the investigation of positional bias to the multimodal setting by empirically examining its existence, patterns, and the underlying causes or contributing factors. 2.3 Multimodal Representation Learning Multimodal representation learning models, particularly CLIP ( 17 ) and its variants, align visual and textual inputs in a shared embedding space using contrastive objectives. While CLIP demonstrates strong performance, it relies on large batch sizes and is limited to short textual inputs. SigLIP ( 25 ) mitigates the batch size issue by replacing the contrastive loss with a sigmoid-based alternative. To address text length limitations, Long-CLIP ( 26 ) introduces a knowledge-preserving positional embedding stretching strategy, extending the token limit to 248 and fine-tuning the model on a dataset specifically curated for long image-text pairs ( 2 ) . In contrast to such approaches that rely on absolute positional encoding, TULIP ( 14 ) adopts relative positional encoding, enabling the model to process text of arbitrary length. This design allows CLIP-like models to make full use of longer captions, thereby enhancing their ability to capture fine-grained and detailed semantic information embedded in extended textual contexts. Other multimodal models, such as LLaVA ( 12 ) and BLIP-2 ( 10 ) , have gained prominence for enabling complex, instruction-based multimodal generation tasks. However, in this work, we focus on CLIP and its variants, which are designed for multimodal representation learning through contrastive alignment of image and text embeddings. This focus is motivated by the fact that CLIP-style models frequently serve as the key backbone for more advanced multimodal systems, including those used in generative settings. In addition, cross-modal retrieval plays a crucial role in multimodal generation pipelines that incorporate Retrieval-Augmented Generation (RAG) techniques ( 22 ; 23 ) . As such, a deeper understanding of CLIP-like models in the context of cross-modal retrieval is critical for advancing

Read the complete text