Mitigating Cross-Image Information Leakage in LVLMs for Multi-Image Tasks

Yeji Park, Minyoung Lee, Sanghyuk Chun, Junsuk Choe · 2025

Read Mitigating Cross-Image Information Leakage in LVLMs for Multi-Image Tasks by Yeji Park, Minyoung Lee, Sanghyuk Chun, Junsuk Choe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13744.txt.

Introduction

Yeji Park and co-authors name and fix a flaw in large models that work with both image and words. When fed many images at once, facts from one image leak into the model's view of the next. Skill drops a lot on multi-image tasks, and the cause was not well mapped before.

Their fix, FOCUS, needs no new training and fits any model shape. It masks all but one image with noise, so the model reads the one clean image. The step is run for each image, and the results are merged, with a noise only pass to clean the blend. Gains held across tests, and the method carried to video as well.

Catalog record

CREATORYeji Park, Minyoung Lee, Sanghyuk Chun, Junsuk Choe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-13744

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mitigating Cross-Image Information Leakage in LVLMs for Multi-Image Tasks Yeji Park, Minyoung Lee, Sanghyuk Chun, Junsuk Choe Abstract Large Vision-Language Models (LVLMs) demonstrate strong performance on single-image tasks. However, we observe that their performance degrades significantly when handling multi-image inputs. This occurs because visual cues from different images become entangled in the model’s output. We refer to this phenomenon as cross-image information leakage . To address this issue, we propose FOCUS , a training-free and architecture-agnostic decoding strategy that mitigates cross-image information leakage during inference. FOCUS sequentially masks all but one image with random noise, guiding the model to focus on the single clean image. We repeat this process across all target images to obtain logits under partially masked contexts. These logits are aggregated and then contrastively refined using a noise-only reference input, which suppresses the leakage and yields more accurate outputs. FOCUS consistently improves performance across four multi-image benchmarks and diverse LVLM families. This demonstrates that FOCUS offers a general and practical solution for enhancing multi-image reasoning without additional training or architectural modifications. Source code is available at https://github.com/yejipark-m/FOCUS Introduction Large Vision-Language Models (LVLMs) are designed to jointly understand visual and textual information ( Li et al. 2023a ; Liu et al. 2023 ; Dai et al. 2023 ; Chen et al. 2024a ; Bai et al. 2025 ; Li et al. 2024a ; Achiam et al. 2023 ) , enabling them to perform a wide range of vision-language tasks, such as Visual Question Answering (VQA) ( Antol et al. 2015 ) and Image Captioning ( Herdade et al. 2019 ) . Figure 1: Illustration of cross-image information leakage in LVLMs during single-image vs. multi-image inference. Although these successes have been largely achieved in single-image settings, challenges still remain when extending these models to multi-image settings, where LVLMs exhibit a notable performance drop ( Wang et al. 2025b ) . We observed that current LVLMs, when given multiple image inputs, often fail to treat each image independently. Instead, they mix visual cues across inputs, leading to a phenomenon we term cross-image information leakage . As illustrated in Figure 1 (a), a model can provide accurate predictions when a single image is given. However, when two images are given simultaneously, as in Figure 1 (b), the model generates incorrect responses that conflate visual elements from both images. Specifically, while image A contains “beer and bananas” and image B contains “banana and coffee container”, the model mistakenly describes image A as “beer, banana, and coffee container”, incorporating irrelevant content from image B, and vice versa. Why does cross-image information leakage happen? We hypothesize that this problem arises since language models (LMs) lack an explicit mechanism to separate visual semantics across different images. LMs often entangle information across inputs, generating outputs that mistakenly include unrelated content from different images. Our goal is to mitigate this cross-image information leakage and to achieve a better multi-image understanding of LVLMs. Multi-image reasoning in LVLMs has been approached through both training-based and inference-based methods, yet key challenges remain in both. Training-based methods ( Awadalla et al. 2023 ; Lin et al. 2024 ; Sun et al. 2024 ; Jiang et al. 2024 ) adopt interleaved image-text sequences to explicitly enhance multi-image reasoning capabilities. However, these methods require massive datasets ( Laurençon et al. 2023 ; Li et al. 2024b ) and computational resources, often hindering practical use and scalability ( Jiang et al. 2024 ) . To avoid expensive training computation and budget, there have been a number of works to improve image understanding in inference-time. For example, Tian et al. 2025 tackled multi-image tasks by adjusting the causal attention mask to reduce position bias. However, this method requires an intensive architecture-level modification that should be tailored to each model. As another line of work, inference-time decoding strategies in LVLM have been studied by editing logits or hidden states during generation ( Leng et al. 2024 ; Chen et al. 2024b ; Park et al. 2025 ) . However, these methods are designed for single-image understanding and are usually not able to be generalized to multi-image tasks. In this paper, we aim to mitigate the cross-image information leakage problem in a resource-efficient manner solely using inference-time operations. Our key idea is to encourage the model to focus on each image individually rather than processing all images simultaneously. To this end, we propose FOCUS, a novel decoding strategy that leverages a noise-guided image focusing technique. As illustrated in Figure 2 , we mask all but one image with random noise, prompting the model to concentrate on the single clean image. We sequentially perform multiple forward passes, each time keeping one image clean, and aggregate the resulting output logits while preserving the positional context of the images. To suppress residual signals from the masked images, we perform an additional forward pass with an input where all images are noise-masked to compute a reference logit, which is subtracted from each output. This process yields a logit distribution that more faithfully reflects the model’s independent understanding of each image. We validate FOCUS on three LVLMs across four multi-image benchmarks: Winoground ( Thrush et al. 2022 ) , VisMin-Bench ( Awal et al. 2024 ) , Mantis-Eval ( Jiang et al. 2024 ) and MuirBench ( Wang et al. 2025b ) . FOCUS achieves consistent improvements, with the best gains of up to +32.1 Image and +29.9 Group score on VisMin, +18.8 Image and +16.8 Group score on Winoground, +5.5%pts accuracy on Mantis-Eval, and +1.5%pts on MuirBench each observed on different model families. These gains are achieved without any additional training or architectural changes, highlighting the effectiveness and generalizability of our method. Related Work While LVLMs have achieved strong performance on standard single-image tasks ( Chen et al. 2024a ; Li et al. 2024a ) , their ability to understand multi-image remains underdeveloped. Until recently, most open-source LVLMs ( Dai et al. 2023 ; Liu et al. 2023 ; Zhu et al. 2024 ; Liu et al. 2024 ) were trained under the assumption of single-image inputs, and therefore struggle to generalize when presented with multiple images simultaneously ( Wang et al. 2025b ) . Prior Work on Multi-Image LVLMs. To address this gap, several recent LVLMs ( Awadalla et al. 2023 ; Sun et al. 2024 ; Laurençon et al. 2023 ; Lin et al. 2024 ; Li et al. 2024b ) have been trained using large-scale interleaved image-text datasets ( Laurençon et al. 2023 ; Li et al. 2024b ) . For example, Laurençon et al. 2023 comprises hundreds of millions of interleaved image-text data that support multiple, contextually related images. While these approaches improve multi-image understanding capabilities, they require extensive training resources ( Jiang et al. 2024 ) and suffer from limited reusability and flexibility due to their reliance on model-specific architectures. Training-free approaches that modify model structure have also been explored. For example, Tian et al. 2025 address position bias in multi-image LVLMs, by altering the causal attention mask used during auto-regressive generation. While effective at reducing positional preference, this method still requires model-specific architectural modifications, which may complicate integration into existing model pipelines. In contrast, we focus on a decoding strategy that requires no changes to

Read the complete text