Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought

Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang Alex Jinpeng Wang Zhi Chen Wanxiang Che Libo Qin · 2025

Read Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought by Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang Alex Jinpeng Wang Zhi Chen Wanxiang Che Libo Qin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15510.txt.

Introduction

Zihui Cheng and co-authors take a unified look at how vision-language models reason step by step about images, in a 2025 paper. Such step-by-step methods, called multimodal chain-of-thought, come in two styles: some reason in text alone, and others mix images into the chain. Both styles boost scores, but why they help was unclear. The team argues that the gain comes from what they call visual thoughts.

Visual thoughts are the image facts a model carries into its reasoning, in whatever form the chain takes. The team defines four forms these thoughts can take and compares how clear and compact each one is. Clearer, tighter forms gave bigger gains in their tests. Probing inside the models, they found visual thoughts ferry image facts to deeper layers of the network. The authors hope the idea guides the design of future reasoning systems.

Catalog record

CREATORZihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang Alex Jinpeng Wang Zhi Chen Wanxiang Che Libo Qin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15510

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang Alex Jinpeng Wang Zhi Chen Wanxiang Che Libo Qin Abstract Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood. To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating visual thoughts , which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ in clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers, enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research. 1 Introduction Recently, Large Vision-Language Models (LVLMs) have made remarkable advancements in tackling various multimodal tasks [ 57 , 22 , 52 ] . Inspired by the success of Chain-of-Thought (CoT) reasoning [ 45 , 17 , 3 ] , LVLMs have integrated Multi-modal CoT (MCoT) reasoning, enabling step-by-step generation of reasoning paths in multimodal contexts. This advancement has enhanced their reasoning abilities, facilitating more sophisticated interactions with multimodal inputs [ 54 , 4 ] . Specifically, current MCoT techniques are generally divided into two paradigms: (1) Textual-MCoT (T-MCoT) : This approach adheres to the traditional CoT framework, generating text-based rationales from multimodal inputs (see Figure 1 (a)). For example, some methods require LVLMs to interpret and describe visual elements before producing an answer [ 55 , 51 ] , while others enhance reasoning by integrating json-format scene graphs derived from images [ 30 , 31 ] . (2) Interleaved-MCoT (I-MCoT) : A more recent approach, such as o3-mini [ 33 ] and Visual Sketchpad [ 15 ] , generates interleaved image-text rationales (see Figure 1 (b)). This method employs external tools, like code interpreters or specialized visual models, to modify images for reasoning [ 28 , 15 , 56 ] , while others use image generation models to create new images, enhancing the reasoning process [ 27 , 19 ] . Figure 1 : Comparison between (a) Textual MCoT (T-MCoT) with purely textual rationale, and (b) Interleaved MCoT (I-MCoT) with the image-text interleaved rationale. VT: visual thoughts. However, the debate between these two paradigms remains unresolved. Some researchers contend that the interleaved rationale in I-MCoT better reflects human cognitive processing of multimodal inputs, potentially offering advantages over T-MCoT [ 15 , 19 ] . In contrast, other studies suggest that in mathematical contexts, text-only rationales may yield superior performance [ 56 , 7 ] . This disagreement highlights a fundamental gap in understanding the mechanisms behind different MCoT approaches. In addition, in the literature, there is a lack of a unified framework to explain MCoT’s effectiveness, identify optimal MCoT paradigms, or derive generalizable insights across tasks. Motivated by this, in this work, we aim to investigate the following research question: Is there a unified explanation for that different MCoT paradigms enhance LVLMs in distinct ways? To this end, we reveal that MCoT unifiedly improves LVLMs by integrating visual thoughts [ 18 ] into the rationale generation. Visual thoughts are intermediate, logic-driven cross-modal representations that facilitate and accelerate multimodal reasoning within a unified perspective. By caching distilled visual information, they bridge raw pixels and linguistic rationales, enabling fast, context-aware access without reprocessing the image. Conceptually, as shown in Figure 2 , image features and visual thoughts function like external memory and cache: raw images require slow, resource-intensive retrieval, while visual thoughts retain key content for rapid access, improving reasoning efficiency and reducing computational cost. Moreover, the effectiveness and efficiency of visual thought transmission can further explain performance differences across MCoT paradigms. To verify this, during experiments, we first verify the effectiveness of visual thoughts in improving MCoT performance. Then, to extensively explore how visual thoughts work in different expressions, we categorize four major strategies: Natural Language , Structured Language , Edited Image , and Generative Image visual thoughts. Moreover, we also investigate the role of internal attention mechanisms and information flow within LVLMs to analyze the rationale behind visual thoughts. Our findings reveal the following: (1) Removing visual thoughts and forcing reasoning solely from the original image can impair performance, even more than reasoning directly from the query. (2) Different expressions of visual thoughts are more effective in certain scenarios, depending on their expression clarity and efficiency. (3) Visual thoughts not only carry visual information but also serve as primary intermediaries, connecting the input image to deeper transformer layers and enabling more advanced cognitive processing in LVLMs. Our main contributions can be summarized as follows: • We present the first comprehensive exploration of the underlying mechanisms driving visual thoughts in MCoT reasoning. This unified perspective provides novel insights into how visual cognition unfolds during decision-making processes. • We introduce four distinct strategies for systematically exploring visual thoughts, demonstrating the advantages and disadvantages of different strategies. • Our analysis further delves deeply into the nature of visual thoughts, revealing critical insights, such as the way in which visual information is integrated into the reasoning path, which enables conveying more visual information into the deeper transformer layers of LVLMs. We hope these findings will inspire more breakthroughs for future research in this area. 2 Visual Thoughts 2.1 Definition of Visual Thoughts To explain how MCoT improves performance, we contend that its primary benefit is the incorporation of visual thoughts , intermediate reasoning steps that explicitly convey visual information and enable LVLMs to perform deeper visual reasoning. As shown in Figure 2 , LVLMs treat the raw image as external memory, forcing the model to iteratively reprocess the entire visual input at each step so as to cap the depth of reasoning. In contrast, visual thoughts extract only the instruction-relevant regions (e.g., the left and right apples) and store them as a cache. Subsequent reasoning steps query this cache rather than the full image, reducing computational overhead and enabling deeper, multi-step MCoT. Formally, a visual thought ( \mathcal{R}^{\mathcal{E}_{\textit{VT}}} ) is a reasoning step that conveys information from the visual input \mathcal{V}_{I} and all previous steps \mathbf{R}_{<i}=\{\mathcal{R}_{1},\mathcal{R}_{2},\dots,\mathcal{R}_{i-1}\} . These steps are driven by the task question \mathcal{Q}_{T} and by explicit instructions \mathcal{I}_{\mathcal{E}} requesting the MCoT expression \mathcal{E}_{\textit{VT}} . The model then generates the next reasoning step \mathcal{R}_{i} as follows: \mathcal{R}_{i}=\begin{cases}\underset{\mathcal{R}^{\mathcal{E}_{\textit{VT}}}}{\operatorname{argmax}}\ \dot{\pi}\left(\mathbf{R}_{<i},\mathcal{V}_{I}\rightarrowtail\mathcal{R}^{\mathcal{E}_{\textit{VT}}}|\mathcal{V}_{I},\mathcal{Q}_{T},\mathcal{I}_{\mathcal{E}_{\textit{VT}}},\mathbf{R}_{<i}\right),&\text{if }\dot{\pi}\geq\pi,\\ \underset{\mathcal{R}^{D}}{\operatorname{argmax}}\ \pi(\mathbf{R}_{<i},\mathcal{R}^{\mathcal{E}_{\textit{VT}}}\rightarrowtail\mathcal{R}^{D}|\mathcal{V}_{I},\mathcal{Q}_{T},\mathcal{I}_{\mathcal{E}_{\textit{VT}}},\mathbf{R}_{<i}),&\text{if }\dot{\pi}<\pi,\end{cases} (1) where \mathcal{R}^{D} represents the derivative reasoning steps [ 58 ] following \mathcal{R}_{i} and grasp visual information from \mathcal{R}^{\mathcal{E}_{\textit{VT}}} . The function \dot{\pi}(\cdot) denotes the probability of generating visual thoughts, and \pi(\cdot) represents the probability of generating derivative reasoning steps. The symbol x\rightarrowtail y indicates the reasoning step y with the flow of reasoning information from

Read the complete text