Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks
Chenjun Li · 2026
Read Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks by Chenjun Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04676.txt.
Introduction
Li decodes the pulse of reasoning VLMs in multi-image understanding tasks. Multi-image reasoning stays hard. During chain-of-thought generation, text-to-image attention shows diffuse pulses missing task-relevant images. A positional bias in attention across images also appears. PulseFocus, a training-free method, structures reasoning into interleaved plan and focus blocks with soft attention gating. Forcing explicit planning of which image to examine, then gating attention to the referenced image, sharpens focus. Gains reach 3.7 percent on the BLINK benchmark, with a smaller gain on MuirBench.
Catalog record
| CREATOR | Chenjun Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17140 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04676
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Decoding the Pulse of Reasoning VLMs in Multi-Image Understanding Tasks Chenjun Li Abstract Note: This is an on-going project. Results and methods are subject to change as experiments continue. Multi-image reasoning remains a significant challenge for vision-language models (VLMs). We investigate a previously overlooked phenomenon: during chain-of-thought (CoT) generation, the text-to-image (T2I) attention of reasoning VLMs exhibits diffuse “pulses”: sporadic and unfocused attention patterns that fail to concentrate on task-relevant images. We further reveal a systematic positional bias in attention allocation across images. Motivated by these observations, we propose PulseFocus , a training-free, inference-time method that structures CoT reasoning into interleaved <plan> / <focus:I> blocks with soft attention gating. By forcing the model to explicitly plan which image to examine and then gating decode-time attention to the referenced image, PulseFocus sharpens attention focus and yields consistent improvements on multi-image benchmarks like BLINK benchmark (+3.7%) and MuirBench (+1.07%). 1 Introduction Recent vision-language models (VLMs) with reasoning capabilites, such as InternVL3.5 [ 15 ] , Qwen3-VL [ 1 ] , and GPT-5 [ 13 ] , have demonstrated impressive single-image understanding. However, multi-image reasoning tasks requiring comparing, counting, ordering, or grounding across multiple images, remains substantially harder [ 14 , 4 , 6 ] . A growing number of benchmarks [ 14 , 4 , 11 , 10 ] has exposed persistent failure modes: VLMs often confuse image identities, exhibit positional bias toward earlier images, and produce hallucinated cross-image comparisons [ 3 ] . Figure 1 : Example case (from MuirBench). Baseline CoT fails to focus on the key evidence image (I5): token-level T2I colouring remains diffuse, and the model cannot recognize the second car. With PulseFocus , the <focus:I5> block becomes consistently image-aligned and the final answer is corrected from (C) to (B). We ask: why do VLMs struggle with multi-image tasks? Rather than attributing failure solely to insufficient training data or model capacity, we investigate the internal attention dynamics during chain-of-thought reasoning. Our analysis reveals two key phenomena: 1. Diffuse T2I attention pulses. During CoT generation, the model’s text-to-image (T2I) attention “pulses” sporadically across all images instead of concentrating on the image currently being discussed (Figure 1 ). This scattered attention correlates with reasoning errors. 2. Positional attention bias. Aggregating attention over many samples reveals systematic bias: earlier images receive more attention regardless of task relevance (Section 3 ). These findings motivate PulseFocus , a training-free inference-time intervention. PulseFocus restructures CoT generation into explicit <plan> / <focus:I> blocks, where the model first plans which image(s) to examine, then generates observations with soft attention gating that amplifies attention to the referenced image while suppressing (but not eliminating) attention to others. This interleaved structure provides two benefits: (1) it forces systematic, image-by-image reasoning instead of ad-hoc jumps between images, and (2) the soft gating sharpens attention focus during each observation, reducing cross-image confusion. We evaluate PulseFocus across three multi-image benchmarks—MuirBench [ 14 ] , BLINK [ 4 ] , and Visual Haystacks [ 16 ] —using InternVL3.5 and Qwen3-VL model families. Our results show that PulseFocus yields notable improvement on BLINK (+3.7% with phase-budget control) and competitive performance on MuirBench, with qualitative analysis confirming sharper attention focus on case studies involving counting and image identity confusion. 2 Related Work Multi-Image VLMs and Benchmarks. Multi-image understanding has been formalized through benchmarks including MuirBench [ 14 ] (12 reasoning capabilities over 2,600 samples), BLINK [ 4 ] (14 perceptual subtasks), MMIU [ 11 ] , MIBench [ 10 ] , and MMSI-Bench [ 18 ] . Model architectures such as InternVL3.5 [ 15 ] and Qwen3-VL [ 1 ] employ interleaved image-text representations with rotary position encoding, while training pipelines like Mantis [ 6 ] and LLaVA-OneVision [ 8 ] use curated multi-image instruction data. Failure Modes in Multi-Image Reasoning. Wei et al. [ 3 ] identify systematic failure modes including image identity confusion and positional bias. Fan et al. [ 3 ] demonstrate that VLMs exhibit position-dependent performance degradation as image count increases. These studies diagnose the problem but do not intervene in the model’s internal attention mechanisms. Attention-Based Interventions. Rethinking causal masking for VLMs [ 12 ] proposes non-causal attention across image tokens, while multi-layer learnable attention masks [ 2 ] train additional mask parameters. V2PE [ 5 ] introduces variable position encoding for better long-context multi-image handling. Unlike these approaches, PulseFocus requires no training and operates purely at inference time through structured prompting and soft attention gating. Divide-and-Conquer Reasoning. IdealGPT [ 19 ] decomposes VQA into iterative sub-questions, while divide-and-conquer approaches [ 9 , 7 ] split complex reasoning into manageable steps. Interleaved reasoning via RL [ 17 ] shows that structured multi-hop reasoning improves over monolithic CoT. CMMCoT [ 20 ] augments CoT with memory for multi-image tasks. PulseFocus shares the spirit of structured reasoning but uniquely combines it with attention-level gating to enforce focus. 3 Attention Analysis of Reasoning VLMs We begin by analyzing the internal attention patterns of VLMs during multi-image CoT generation. All experiments in this section use InternVL3.5-8B on MuirBench [ 14 ] . 3.1 Text-to-Image Attention Pulses During autoregressive CoT generation, we extract text-to-image attention weights at each decode step. For a generated token t_{k} , we compute the attention mass allocated to each image I_{j} : a_{k,j}=\sum_{p\in\mathcal{S}_{j}}\alpha_{k,p} (1) where \mathcal{S}_{j} is the set of visual token positions for image I_{j} and \alpha_{k,p} is the attention weight from token k to position p , averaged across selected attention heads at layer depths of 0%, 50%, and 100%. Observation 1: Scattered pulses. Plotting a_{k,j} over decode steps reveals that T2I attention “pulses” (brief spikes in attention to image tokens) are not well-aligned with the image being discussed. When the model’s CoT text references “Image 2,” the attention often spreads nearly uniformly across all images rather than concentrating on Image 2. We term this the scattered pulse phenomenon (Figure 2 ). Figure 2 : Attention pulse visualization. T2I attention mass per image over CoT decode steps for a counting task (the same example as in Figure 1, with six input images). Top: baseline—attention is spread across images even when discussing a specific image. Bottom: with PulseFocus —attention concentrates on the image specified in the <focus:I> block. Colors indicate different images. 3.2 Positional Attention Bias Aggregating T2I attention across all 2,600 MuirBench samples, we examine how attention distributes across image positions. Observation 2: Earlier images receive more attention. Figure 3 shows the mean attention mass per image position, aggregated across samples with 2–6 images. Earlier image positions (I1, I2) consistently receive higher attention than later positions, independent of task type. This positional bias mirrors findings by Fan et al. [ 3 ] but is measured here at the internal attention level rather than through accuracy proxies. Figure 3 : Positional attention bias. Mean T2I attention mass per image position for InternVL3.5-8B on MuirBench. Earlier images receive disproportionately more attention regardless of task. Error bars show standard deviation across task types. 4 Method: PulseFocus Motivated by the scattered pulse and positional bias findings, we propose PulseFocus , which combines structured interleaved prompting with soft attention gating at decode time. Figure 4 provides an overview of the full pipeline. Figure 4 : PulseFocus overview. The model alternates between <plan> blocks