Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning
Yang Zhang, Xiaoshuai Sun, Rui Zhao, Wujin Sun, Yidong Chen, Jiayi Ji, Qian Chen, Rongrong Ji · 2026
Read Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning by Yang Zhang, Xiaoshuai Sun, Rui Zhao, Wujin Sun, Yidong Chen, Jiayi Ji, Qian Chen, Rongrong Ji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28160.txt.
Introduction
Zhang and colleagues ask when a reasoning model should actually look at an image instead of converting it to text up front or reasoning in one fused space. Their CSMR framework lets the language model call an independent vision module on demand, fetching task-relevant evidence mid-thought. Across multimodal benchmarks it beat representative baselines in zero-shot accuracy, thanks to that scheduling.
Catalog record
| CREATOR | Yang Zhang, Xiaoshuai Sun, Rui Zhao, Wujin Sun, Yidong Chen, Jiayi Ji, Qian Chen, Rongrong Ji |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43957 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28160
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning Yang Zhang, Xiaoshuai Sun, Rui Zhao, Wujin Sun, Yidong Chen, Jiayi Ji, Qian Chen, Rongrong Ji Abstract Existing multimodal reasoning approaches mainly follow two paradigms: pre-reasoning visual-to-text conversion or reasoning in a unified vision–language space. The former compresses fine-grained visual details through static textualization, while the latter suffers from linguistic dominance, weakening faithfulness to visual evidence. In this work, we argue that a central challenge is how and when visual evidence is introduced into the reasoning process. Motivated by this insight, we propose a multimodal reasoning framework in which a language model maintains a reasoning state and dynamically schedules visual evidence acquisition, deciding both when to query an independent perception module and when to terminate reasoning. Experiments across multiple multimodal reasoning benchmarks show that CSMR consistently outperforms representative baselines in accuracy under zero-shot settings. Further analysis demonstrates that these gains stem from reasoning-state-driven visual querying and early termination. The code is available at https://github.com/YangZhang2511/CSMR 1 Introduction In recent years, the rapid advancement of large-scale Vision–Language Models (VLMs) has significantly accelerated research on multimodal reasoning. A central goal of this line of work is to enable models to perform complex reasoning by effectively integrating visual and linguistic information. Depending on how visual evidence is introduced during reasoning, existing approaches can be broadly categorized into two paradigms. The first paradigm converts visual inputs into textualized visual evidence before reasoning, and subsequently performs the entire reasoning process purely in the language space ( 39 ; 31 ; 44 ) , as illustrated in Fig. 1 (a). This design directly leverages the strong reasoning capabilities of large language models (LLMs). However, since the reasoning trajectory has not yet unfolded at the time of one-shot visual-to-text conversion, the initial textualization can only capture coarse semantics, inevitably compressing or discarding fine-grained evidence that may become decisive at later stages ( 41 ; 3 ) . The second paradigm introduces visual evidence directly into the reasoning process at the representation level ( 28 ; 19 ; 16 ; 27 ) , enabling end-to-end reasoning within a unified vision–language embedding space, as illustrated in Fig. 1 (b). By avoiding explicit visual-to-text conversion, this paradigm retains direct access to raw visual representations, thereby preventing the irreversible loss of fine-grained visual evidence, and has become a prevalent approach in recent multimodal reasoning research. However, as discussed in Section 4 , this paradigm exhibits a structural bias that causes visual representations to be influenced by linguistic priors, resulting in weakened grounding in image evidence. Figure 1 : Illustration of two dominant multimodal reasoning paradigms and our framework. Taken together, these observations suggest that the two paradigms correspond to two typical failure modes: either important visual evidence is lost during the conversion of visual inputs into text, or visual representations are undermined in their faithfulness to image evidence. These observations indicate that the central challenge in multimodal reasoning lies in determining when and how visual evidence should be introduced and integrated into the reasoning process. Inspired by the working memory theory of Baddeley et al. ( 1 ) , we propose CSMR, a cognitive scheduling framework for visual evidence acquisition in multimodal reasoning. As illustrated in Fig. 1 (c), an LLM serves as the cognitive core that controls the reasoning process and dynamically invokes a VLM during inference to obtain necessary visual evidence. Because the acquisition of visual evidence is explicitly driven by the reasoning state, it is no longer treated as a one-time conversion, but rather as a process that can be repeatedly invoked and progressively validated. Even if the visual evidence returned by the perception module is insufficient, the reasoning core can continue to invoke the perception module until sufficient visual evidence is obtained to support the reasoning process. Meanwhile, the structural decoupling of perception and reasoning prevents the extracted visual evidence from being influenced by linguistic priors. Our contributions are threefold: 1.We analyze a structural limitation in unified multimodal reasoning: visual representations are influenced by linguistic priors, resulting in weakened grounding in image evidence. 2.We propose CSMR, in which a language model maintains an explicit reasoning state and uses it to govern the reasoning process, dynamically invoking an independent visual perception module to acquire task-relevant visual evidence. 3.We conduct extensive evaluations on multiple multimodal reasoning benchmarks, demonstrating consistent improvements in accuracy over representative baselines. 2 Related Work 2.1 Text-Centric Pre-Reasoning Paradigm In this paradigm, visual inputs are converted into textual representations prior to reasoning, and all subsequent reasoning is performed purely in the language space without further access to the original images. Early approaches rely on global image descriptions for vision-to-text conversion ( 39 ; 31 ) , while later works improve task alignment by introducing question-relevant captions ( 12 ) or more abstract intermediate textual representations, such as candidate-answer-based reasoning ( 32 ) . DDCoT ( 44 ) further decomposes problems into sub-questions, queries visual evidence via external tools, and integrates the resulting textual evidence for final reasoning. Despite their effectiveness, these approaches rely on static and pre-planned textualization of visual inputs, preventing the model from incrementally supplementing or revising visual evidence during reasoning, which limits performance on tasks requiring fine-grained or iterative visual verification. 2.2 Unified Multimodal Representation Paradigm This paradigm performs end-to-end reasoning by fusing visual and textual features into a unified multimodal representation space. By jointly modeling both modalities, these methods aim to enhance cross-modal interaction during reasoning. Existing approaches can be roughly grouped into three categories. Early methods, such as T5 ( 29 ) , fuse visual and textual features but generate text-only reasoning outputs. Subsequent works introduce image-related intermediate structures or prompting signals to guide reasoning in the shared space, including CCoT ( 28 ) and SCAFFOLD ( 16 ) . More recent methods further intensify multimodal interaction by jointly generating visual and textual information during reasoning, enabling multimodal reasoning chains, as exemplified by ICoT ( 10 ) and AIMCoT ( 19 ) . Despite these advances, this paradigm is prone to hallucinations ( 9 ; 23 ; 36 ; 38 ) ; this paper further analyzes the underlying causes of this issue. As analyzed in Section 4 , under joint multimodal training objectives, dominant linguistic priors may bias visual representations toward the language space, undermining faithfulness to the original image evidence. In addition, some interactive or tool-augmented reasoning approaches ( 6 ; 40 ; 11 ) mainly reason by imitating demonstration trajectories, making them essentially trajectory imitation-driven. Thus, both query generation and answer prediction tend to follow demonstrated patterns rather than being guided by the internal reasoning state of the current task. 3 Preliminaries In this paper, we focus on the single-stream paradigm, which is the dominant architecture in contemporary VLMs such as LLaVA ( 17 ) and Qwen-VL ( 35 ) . A typical single-stream VLM consists of a vision encoder E_{v} and a decoder-only LLM, which process visual and textual information within a unified token sequence. Given an image I and a textual question q , the vision encoder encodes the image into visual embeddings: x^{\text{vis}}=E_{v}(I). (1) The question is converted into an instruction-style prompt \text{Prompt}_{\mathrm{instruct}} and encoded into textual embeddings: x^{\text{text}}=E_{t}\!\left(\text{Prompt}_{\mathrm{instruct}}(q)\right). (2) where E_{t}(\cdot) denotes