AIMCoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

Xiping LI, Jianghong Ma · 2025

Read AIMCoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning by Xiping LI, Jianghong Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25699.txt.

Introduction

Xiping Li and Jianghong Ma improve how vision-language models reason step by step with image evidence, in a paper from 2025. Strong reasoning inserts well-chosen image details into the thought chain at the right moments. Prior methods picked evidence from attention signals, which can mislead, and inserted it on a fixed schedule.

Their framework tackles both flaws. Text context first sharpens the attention map, then an active search picks the most useful evidence, much like foraging. A trigger watches for attention shifts from text to image and inserts evidence at those moments. Across three benchmarks and four model backbones, results beat prior methods consistently.

Catalog record

CREATORXiping LI, Jianghong Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25699

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AIMCoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning Xiping LI, Jianghong Ma Abstract Multimodal Chain-of-Thought (CoT) has emerged as a powerful technique for enhancing the vision-language reasoning with interleaved information. However, existing methods often rely on simplistic heuristics for constructing interleaved CoT, typically depending on attention maps, which our empirical analysis reveals can be unreliable. What’s more, the shortcomings of their passive and purposeless selection strategies and their arbitrary triggering mechanisms in capturing the model’s cognitive need for information are further amplified. In this paper, we propose AIMCoT , an A ctive I nformation-driven M ulti-modal C hain- o f- T hought framework that addresses these fundamental limitations. AIMCoT introduces three synergistic components: (1) Context-enhanced Attention-map Generation (CAG) , which mitigates the text-vision granularity imbalance, thereby producing more reliable attention maps as a foundation. (2) Active Visual Probing (AVP) , which replaces passive selection with a proactive, goal-oriented strategy grounded in information theory to select image regions that help answer the questions maximally. (3) Dynamic Attention-shifting Trigger (DAT) , which intelligently determines the optimal moments to insert visual information by monitoring the model’s text-to-vision attention shifts. Extensive experiments on three challenging benchmarks demonstrate that AIMCoT significantly outperforms state-of-the-art methods across different settings. By actively foraging for information and dynamically structuring its reasoning process, AIMCoT represents a critical step towards more robust, effective, and human-like multimodal reasoning. Our code is available at https://anonymous.4open.science/r/AIMCoT. 1 Introduction The advent of Chain-of-Thought (CoT) prompting has marked a significant milestone in the reasoning capabilities of Large Language Models (LLMs) Wei et al. 2022 ; Wang et al. 2022 ; Zhang et al. 2022 ; Suzgun et al. 2022 ; Li et al. 2025 ; Yao et al. 2023 ; Besta et al. 2024 , enabling them to deconstruct complex problems into a series of intermediate, interpretable steps. This paradigm has been naturally extended to Vision-Language Models (VLMs), where early efforts Mitra et al. 2024 ; Zheng et al. 2023 ; Lei et al. 2024 ; Zhang et al. 2023 focused on generating text-only rationales to articulate the model’s reasoning process over visual inputs. A pivotal advancement in this domain was the introduction of Interleaved-modal Chain-of-Thought Gao et al. 2025 , which pioneered the direct integration of visual patches into the reasoning chain. By pairing textual rationales with corresponding image regions, the interleaved CoT demonstrated a superior ability to ground language in visual evidence, setting a new standard for multimodal reasoning. However, the efficacy of existing research Ge et al. 2025 ; Gao et al. 2025 is fundamentally predicated on the reliability of their underlying mechanisms for selecting and integrating visual information. These methods typically rely on a passive, attention-driven strategy: they select the Top-K regions with the highest attention scores and insert them at predefined moments, e.g., the appearance of a newline character, a practice with little theoretical or empirical justification for its timing. This reliance exposes critical vulnerabilities that limit their full potential. Our empirical analysis reveals that high-attention regions are often redundant or, more alarmingly, fail to capture the most crucial visual details, especially when a granularity mismatch exists between the textual query and the visual evidence. This raises three fundamental questions: (1) What information source can reliably identify truly salient visual regions? (2) How can we accurately select useful regions in a proactive and purposeful manner rather than passively relying on potentially unreliable attention scores? (3) When is the optimal moment to insert visual evidence into the reasoning process? In this work, we postulate that a more robust multimodal reasoning framework requires a shift from passive attention-following to an active, information-seeking paradigm. We propose AIMCoT: A ctive I nformation-driven M ulti-modal C hain- o f- T hought, a novel framework that directly addresses the limitations of prior work, reframing the selection of visual evidence from a passive, attention-based retrieval task to an active, goal-oriented probing process. Inspired by the principles of information foraging Pirolli & Card 1999 ; Broadbent 2013 , AIMCoT operates on the premise that the most valuable visual rationale is one that maximally reduces the model’s uncertainty about the subsequent step in its reasoning chain. Rather than simply asking "Where is the model looking?", we prompt the model to actively ask "Which piece of visual information will be most helpful for me to see right now ?". This shift is realized through three synergistic components: 1. Context-enhanced Attention-map Generation (CAG) , which mitigates the text-vision granularity disparity to improve the reliability of attention map for better identifying salient regions by generating a context-aware description of the image. 2. Active Visual Probing (AVP) , which implements a proactive, goal-oriented selection strategy grounded in information theory, selecting a set of visual regions that provide the highest possible information gain for the task at hand. 3. Dynamic Attention-shifting Trigger (DAT) , an intelligent triggering mechanism that carefully captures the critical moments when model’s cognitive focus shifts significantly from text to vision, and inserts visual information precisely. Our contributions are as follows: • We introduce AIMCoT, a novel training-free framework that reframes the construction of multimodal CoT as an active information-foraging process, moving beyond the limitations of static, passive, and purposeless region selection. • We propose a system comprised of three complementary methods (CAG, AVP, DAT) that collectively enable VLMs to proactively forage for informative visual evidence and strategically integrate it into their reasoning process at the detected critical moments. • We present a comprehensive set of empirically-grounded motivations that inspire the designs within our AIMCoT framework. Furthermore, we provide substantial theoretical analysis and design targeted experiments to explore the properties of these designs, covering key aspects such as the deployability of AIMCoT, the interplay between the CAG and AVP modules, and the necessity of incorporating an exploratory candidate pool. • Through extensive experiments conducted on two backbones (Chameleon-7B and Qwen2-VL-7B) across challenging benchmarks including M3CoT, ScienceQA, and LLaVA-W, we demonstrate that AIMCoT significantly outperforms state-of-the-art baselines and advances the frontier of vision-language reasoning. 2 Related Work The success of CoT prompting in LLMs has naturally extended to VLMs, aiming to make their reasoning processes explicit and interpretable. Early efforts focused on generating text-only rationales. MMCoT Zhang et al. 2023 first generates a rationale from the input and then uses this rationale along with the original multimodal data to infer the final answer. CCoT Mitra et al. 2024 prompts the VLM to generate an intermediate scene graph to structure its understanding. DDCoT Zheng et al. 2023 decomposes complex problems into simpler sub-questions and leverages external models to fill information gaps. Alternatively, SCAFFOLD Lei et al. 2024 overlays a coordinate grid on the image, enabling the VLM to reference spatial regions explicitly in its textual reasoning. A pivotal advancement is the introduction of interleaved-modal CoT, which integrates visual evidence directly into the reasoning chain. The leading approach, ICoT Gao et al. 2025 , selects the top-K regions from the VLM’s attention map and inserts them at predefined moments. MRFD Ge et al. 2025 meticulously selects the salient visual regions to foster more reliable reasoning, alleviating VLM hallucination. However, our analysis reveals that such passive, attention-driven strategies are fundamentally limited by the unreliability of raw attention maps, especially during text-vision granularity mismatches, and their arbitrary

Read the complete text