Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning

2026

Read Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26737.txt.

Introduction

Guo, Lu, Feng, and Sun propose SSV-CoT, a visual reasoning method that looks at an image in a planned order. Multimodal models usually freeze the image into static tokens and reason only in text. Human sight instead shifts focus from the most telling region to weaker cues. A saliency map first ranks the regions that matter for the question. Reasoning then follows that order, building a curriculum-like path from primary to secondary evidence. Trained end to end, the method gains across varied visual reasoning tests.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37218
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning Abstract Current multimodal LLMs encode images as static visual prefixes and rely on text-based reasoning, lacking goal-driven and adaptive visual access. Inspired by human visual perception—where attention is selectively and sequentially shifted from the most informative regions to secondary cues—we propose Structural Sequential Visual CoT (SSV-CoT) . First, a question-relevant saliency map identifies and organizes key visual regions, explicitly modeling the spatial distribution of visual importance. Second, reasoning is performed following this discriminative order, inducing a curriculum-like semantic progression from primary to secondary cues. This method is trained end-to-end, using text cot and answer supervision, without relying on region-level annotations or specialized external tools. Experiments on diverse visual reasoning benchmarks show gains, validating structured and sequential visual cognition. 1 Introduction Despite rapid advances in multimodal LLMs ( Alayrac et al. 2022 ; Liu et al. 2023 ; Li et al. 2023 ; Dai et al. 2023 ) , current systems still suffer from a fundamental limitation in how vision is integrated into reasoning. Most architectures encode an image once into visual tokens or a global embedding and inject it into the LLM as static context ( Alayrac et al. 2022 ; Li et al. 2023 ; Liu et al. 2023 ) . Reasoning then unfolds purely in text space: as the chain-of-thought grows, the model increasingly ignores the image, cannot revisit or operate on visual content, and lacks goal-directed visual actions such as selecting, refocusing, or comparing regions. As a result, visual perception and language reasoning remain largely disentangled, and vision functions as a passive prefix rather than an active part of cognition. Existing attempts to mitigate this gap face significant drawbacks. External tool pipelines introduce cropping or detection but require complex orchestration ( Wu et al. 2023 ; Chen et al. 2023 ) . Region-based supervision depends on costly annotations ( Lin et al. 2022 ; Khan et al. 2022 ) . Generative visual-token or image synthesis is inefficient and poorly aligned with reasoning needs ( Esser et al. 2021 ; Ding et al. 2022 ) . Latent-space alignment methods learn global correspondences yet provide no explicit visual access steps ( Radford et al. 2021 ; Li et al. 2023 ) . Interleaved chain-of-thought methods re-inject visual tokens but enable passive re-attention over a large patch space, without modeling where to look next ( Zhang et al. 2024b ) . None of these approaches support internal, sequential, goal-conditioned visual cognition. Figure 1: MLLM lacks visual analysis, while SSV integrates cues through structured sequential reasoning to reach the correct answer. As illustrated in Fig. 1 , humans rarely solve visual tasks by processing an entire scene uniformly. Instead, they first form a coarse global impression and then sequentially attend to task-relevant regions as reasoning unfolds, a pattern well documented in studies of human visual attention and eye movements ( Yarbus 1967 ; Itti et al. 1998 ) . In the example, the left approach (MLLM-CoT) fixates on the presence of snow-covered ground and ski equipment, ignores critical cues such as strong sunlight and sharp shadows, and arrives at an incorrect conclusion. In contrast, the right approach (SSV-CoT) explicitly guides visual attention through a structured sequence: it first focuses on salient regions (e.g., ski equipment and shadows) and then integrates additional cues (e.g., lighting conditions) to refine the hypothesis, ultimately reaching the correct answer. This comparison highlights a fundamental limitation of existing multimodal large language models, which typically compress images into a fixed set of visual tokens and perform reasoning purely in text space ( Alayrac et al. 2022 ; Li et al. 2023 ) . While recent work on chain-of-thought reasoning has shown the benefits of explicit intermediate reasoning in language models ( Wei et al. 2022 ) , visual perception in current MLLMs remains largely passive and decoupled from the reasoning process. This disconnect between perception and cognition motivates our central question: can a model learn not only what to look at, but also when and in which order to look, driven directly by its ongoing reasoning? Inspired by human visual cognition, we propose SSV-CoT , a multimodal reasoning framework that couples structured visual representation with sequential, goal-conditioned visual access. Given an image–question pair, SSV-CoT performs a single forward pass through a vision encoder and MLLM to construct a structured visual representation: a question-aware saliency map is generated, binarized, and decomposed via connected component analysis, yielding a bank of candidate region embeddings together with a global scene embedding. Conditioned on the evolving reasoning state, a visual cognition policy scores candidate regions and selects one region (or the global embedding) at each step, injecting its embedding into the MLLM, so that reasoning unfolds as an interleaved process of visual querying and textual inference. To reflect human attentional control, SSV-CoT further incorporates an adaptive stopping mechanism that dynamically determines whether to continue visual access, balancing evidence sufficiency against redundant queries. The method is trained end-to-end from image–question—text cot-answer without gaze traces, bounding boxes, or external tools: a heuristic supervised stage initializes the policy by imitating a question-conditioned saliency ordering derived from the pretrained MLLM, followed by a GRPO reinforcement learning stage that optimizes visual selection using answer-level rewards with a visual-budget regularizer. Overall, SSV-CoT replaces one-shot visual token injection with a structured, sequential, and adaptive visual reasoning process aligned with human cognitive principles. Our main contributions are three-fold: • We model visual access in multimodal reasoning as a human-inspired, goal-conditioned process over structured visual units, rather than passive visual token injection. • We propose a training framework that learns visual selection order from task-level feed- back using text-cot and answer supervision, without region-level annotations. • We introduce an adaptive, budget-aware visual reasoning mechanism that decides when to continue or stop visual querying during chain-of-thought reasoning. 2 Related Work 2.1 Visual Reasoning A broad class of methods augments MLLMs with external visual tools for cropping, detection, or region selection ( Schick et al. 2023 ; Chen et al. 2023 ; Yang et al. 2023 ; Aissi et al. 2025 ) , but these rely on complex, non-differentiable pipelines with manually designed actions decoupled from internal reasoning. Another line of work introduces region-level supervision via bounding boxes, captions, or grounding annotations ( Li et al. 2022 ; Yang et al. 2022 ; Kamath et al. 2021 ; Liu et al. 2024a ) , which improves controllability but requires costly labeling and exposes all regions upfront, preventing sequential selection. Related segmentation-based or region-level abstractions, such as SAM-derived masks or region-level ViTs ( Kirillov et al. 2023 ; Chen et al. 2022 ; Ravi et al. 2024 ) , provide structured features but are typically used as static inputs rather than a decision space. Overall, these approaches treat vision as fixed or externally controlled and do not enable internal, learnable, sequential visual cognition in MLLMs. 2.2 Interleaved Visual Reasoning Interleaved chain-of-thought methods reintroduce visual tokens during reasoning to alleviate image forgetting ( Zhang et al. 2024b ; Chen et al. 2025a ; Shao et al. 2024 ; Zhang et al. 2024a ; Li and Ma 2025 ) . While effective for multi-hop visual question answering and chart reasoning, these approaches repeatedly attend to the same static

Read the complete text