Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration
Mingtao Xian, Yifeng Yang, Qinying Gu, Xinbing Wang, Nanyang Ye · 2026
Read Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration by Mingtao Xian, Yifeng Yang, Qinying Gu, Xinbing Wang, Nanyang Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11591.txt.
Introduction
Xian and four others propose logit-attention divergence calibration against position bias in multi-image search. Multimodal large language models perform strongly in multi-image search yet suffer position bias, with predictions driven by input order over meaning. The authors find output logits heavily biased while internal attention maps stay aligned with visual evidence. Their training-free method uses attention signals for instance fix at run time, needing only a small tuning set. On MS-COCO-based benchmarks it greatly lifts order steadiness and beats rivals by over 40 percent.
Catalog record
| CREATOR | Mingtao Xian, Yifeng Yang, Qinying Gu, Xinbing Wang, Nanyang Ye |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57387 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11591
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Logit-Attention Divergence: Mitigating Position Bias in Multi-Image Retrieval via Attention-Guided Calibration Mingtao Xian, Yifeng Yang, Qinying Gu, Xinbing Wang, Nanyang Ye Abstract Multimodal Large Language Models (MLLMs) have shown strong performance in multi-image cross-modal retrieval, yet suffer from severe position bias, where predictions are dominated by input order rather than semantic relevance. Through empirical analysis, we identify a phenomenon termed Logit-Attention Divergence, in which output logits are heavily biased while internal attention maps remain well-aligned with relevant visual evidence. This observation reveals a fundamental limitation of existing logit-level calibration methods such as PriDe. Based on this insight, we propose a training-free, attention-guided debiasing framework that leverages intrinsic attention signals for instance-level correction at inference time, requiring only a minimal calibration set with negligible computational overhead. Experiments on MS-COCO-based benchmarks show that our method substantially improves permutation invariance and achieves state-of-the-art performance, enhancing accuracy by over 40% compared to baselines. Code is available at https://github.com/brightXian/LAD . † † affiliationnotice: Project lead 1 Introduction Recent advances in Multimodal Large Language Models (MLLMs) have extended cross-modal reasoning to multi-image scenarios ( 9 ; 17 ; 11 ; 24 ) . A critical task is in-context cross-modal retrieval, where models must identify and retrieve target images from sequences containing numerous distractors ( 6 ; 24 ) . This capability is crucial for real-world applications such as e-commerce recommendation ( 27 ; 20 ) , visual search in photo albums ( 32 ) , and content moderation ( 7 ) . Figure 1 : Confusion matrices for LLaVA-OneVision on the multi-image retrieval task ( N=8 , Random setting). Cell values represent the selection rate (%), where darker blue indicates a higher selection rate. The Vanilla baseline (Left) shows vertical stripes due to severe position bias, while our method (Right) restores the diagonal pattern, demonstrating that predictions correctly align with the ground truth. However, MLLMs exhibit severe position bias where predictions are dominated by input order rather than semantic relevance ( 19 ; 21 ) . As shown in Figure 1 (left), on an 8-candidate retrieval task, the model irrationally prefers certain positions (e.g., 48.96% selection rate at Position 7) while ignoring others (0.00% at Position 4). This bias leads to a drastic divergence in performance where accuracy is inflated to 98.87% when the ground truth coincides with the preferred Position 7, yet it collapses to 0.00% if the ground truth falls into the neglected Position 4. This position bias stems from the autoregressive architecture inherited from Large Language Models ( 30 ; 21 ) . While temporal positional encodings ( 18 ; 23 ) and causal attention masks ( 14 ; 22 ) preserve logical order during text generation, applying this architecture to non-temporal visual sequences introduces unintended asymmetries that render predictions highly sensitive to image input ordering due to structural position dependency. Existing training-free interventions for position bias are predominantly developed for text-only LLMs ( 12 ; 33 ; 16 ; 28 ) . However, transferring these paradigms to MLLMs remains challenging as they overlook cross-modal interactions between vision and language. Our analysis in § 3.1 reveals that position bias in MLLMs is not a static offset but a conditional bias that shifts dynamically with target location. This property renders existing methods ineffective: prompt-based approaches ( 35 ; 29 ) fail to adapt to such dynamic behavior, while static statistical calibration methods like PriDe ( 35 ) rely on fixed priors that ignore the visually conditioned nature of position bias in MLLMs. In § 3.2 , we investigate the cross-modal interaction mechanisms of MLLMs and uncover a distinctive phenomenon: Due to position bias, MLLMs often “look at” the correct answer but “say” the wrong one. We term this discrepancy Logit–Attention Divergence . Concretely, while the final output logits are heavily corrupted by positional priors, the model’s internal attention weights remain semantically aligned with the target visual region. Motivated by this insight, we propose the Attention-Guided Debiasing framework, a training-free framework that uses hierarchical attention to correct position bias at inference time. By bridging static positional priors with dynamic visual evidence, our method effectively corrects position bias with only 5 calibration samples and minimal additional computational cost. As shown in Figure 1 (right), it restores the diagonal confusion matrix, recovering permutation invariance. In summary, our contributions are as follows: • We identify a phenomenon termed Logit-Attention Divergence , where structural decoding priors override correct internal attention, causing models to “look right but speak wrong” in multi-image contexts. • We propose Attention-Guided Debiasing, a training-free framework utilizing intrinsic visual signals to rectify position bias, requiring only a minimal calibration set (e.g., 5 samples) with negligible computational overhead. • We construct an adversarial benchmark with visually similar hard negatives to test robustness under challenging conditions. Extensive experiments demonstrate state-of-the-art performance, effectively restoring permutation invariance across both random and adversarial settings, surpassing baselines by over 40% in accuracy. Figure 2 : Logit distributions for candidate answer tokens conditioned on ground truth (GT) positions on LLaVA-OneVision. Each subplot shows a different GT position. Green curves represent mean logit profiles, gray lines show individual samples, and dashed lines mark the GT (green: correct; red: incorrect). Individual samples cluster tightly around the mean within each GT category. 2 Related Work Cross-modal Retrieval. Early approaches to cross-modal retrieval rely on image–text embedding alignment, mapping both modalities into a shared space and performing retrieval via similarity matching (e.g., CLIP ( 13 ) , ALIGN ( 5 ) , SigLIP ( 34 ) ). Although effective for coarse-grained matching, this approach struggles with fine-grained semantics as multimodal information is compressed into a single representation. With the advent of MLLMs, 6 introduced a novel generative retrieval paradigm that represents images as discrete token IDs, enabling MLLMs to memorize and directly generate the target image ID in response to a textual query. Building on this setting, 24 generalized the setting by instantiating retrieval as multi-image caption matching, where the MLLM is prompted to select the image that best matches a given caption by predicting a discrete index token. Position Bias in LLMs. Similar to the position bias observed in LLMs ( 35 ; 3 ; 25 ) , recent work ( 24 ) reveals that MLLMs also suffer from severe position dependency in multi-image caption matching. Specifically, MLLMs show a systematic preference for images placed at specific positions, particularly those appearing at the beginning or the end of the input sequence. Prior studies have attempted to mitigate this bias using explicit debiasing instructions ( 35 ) appended to the system prompt, as well as Chain-of-Thought prompting ( 29 ) (e.g., “Let’s think step by step”). However, experimental results indicate that neither approach effectively alleviates selection bias, suggesting that it is an inherent behavioral property of LLMs rather than an artifact that can be eliminated through simple prompt engineering. Inference-Time Calibration. Inference-time calibration aims to adjust model behavior post hoc to correct model internal biases without modifying model parameters. Prior work generally falls into three categories. (1) Ensemble-based methods, such as self-consistency ( 26 ) , aggregate predictions across multiple sampled outputs to improve robustness, implicitly assuming that biases cancel out through averaging. (2) Statistical calibration methods, like PriDe ( 35 ) , introduce a global static