More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models

Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Fabian Waschkowski, Lukas Wesemann, Peter Tu, Jing Zhang, · 2025

Read More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models by Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Fabian Waschkowski, Lukas Wesemann, Peter Tu, Jing Zhang, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-25848.txt.

Introduction

Xinyu Tian and co-authors study a trade-off in vision-language models in a paper from two thousand twenty-five. Reasoning helps such models solve hard problems that mix images and text. But long chains of thought can also pull the model away from what the image actually shows.

The authors call this drift visual forgetting, where longer reasoning leads to failures on simple visual questions. To counter it, they propose a training method that steers reasoning toward paths grounded in the image. Their resulting model reached top scores on a wide range of benchmarks. The paper argues that reasoning in these models has a dual nature, both a strength and a risk.

Catalog record

CREATORXinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Fabian Waschkowski, Lukas Wesemann, Peter Tu, Jing Zhang,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-25848

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models Xinyu Tian, Shu Zou, Zhaoyuan Yang, Mengqi He, Fabian Waschkowski, Lukas Wesemann, Peter Tu, Jing Zhang, Abstract Reasoning has emerged as a pivotal capability in Large Language Models (LLMs). Through Reinforcement Learning (RL), typically Group Relative Policy Optimization (GRPO), these models are able to solve complex tasks such as mathematics and code generation. Building on these advances, recent research has sought to extend reasoning to Vision-Language Models (VLMs), yielding promising results across diverse visual tasks. Despite this progress, our study uncovers the dual nature of multimodal reasoning: while it substantially enhances logical inference and facilitates performance on challenging problems, it may gradually impair perceptual grounding, leading to recognition failures on otherwise basic visual questions. Through further analysis, we attribute this phenomenon to visual forgetting, wherein prolonged reasoning causes the model to increasingly disregard visual input. To address this, we propose Vision-Anchored Policy Optimization ( VAPO ), a simple yet effective method that explicitly steers the reasoning process toward visually grounded trajectories. Our result model, VAPO-Thinker-7B , significantly strengthens the model’s reliance on visual information and achieves new state-of-the-art results on a wide range of established benchmarks. Project page: https://xytian1008.github.io/VAPO/ . 1 Introduction Reasoning has long been recognized as an essential capability of Large Language Models (LLMs). Early approaches, such as chain-of-thought ( Wei et al. 2022 ; Kojima et al. 2022 ) , encourage models to produce step-by-step explanations before arriving at an answer, whereas more recent advances have introduced aha moments ( Guo et al. 2025 ; Muennighoff et al. 2025 ) , characterized by cognitive behaviors such as self-reflection and verification. These developments have enabled models to address increasingly complex problems, e.g . , mathematics and coding. Consequently, the community has recently sought to integrate reasoning into Vision-Language Models (VLMs). By leveraging Reinforcement Learning (RL) such as Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) , researchers have trained models to think while tackling challenging multimodal tasks, e.g . , geometry or navigation ( Yang et al. 2025c ; Tian et al. 2025b ) . Remarkably, this integration has also demonstrated promising results on traditional visual tasks, e.g . , classification and detection ( Liu et al. 2025b ; Shen et al. 2025 ) , while yielding stronger generalization ( Chu et al. 2025 ) . Building on these successful practices, reasoning has emerged as a powerful and seemingly universal strategy for addressing a wide range of visual tasks. However, its limitations remain largely unexplored. Early doubts are raised by Li et al. 2025 , which find that in certain scenarios, models trained with explicit reasoning result in only marginal gains compared to direct answering. More recently, Xia et al. 2025 observe a counter-intuitive trend: unlike in text domains, RL in multimodal settings often exhibits a collapse. As accuracy improves during training, the reasoning process gradually shortens, suggesting that models tend to favor brief, less structured responses. These findings lead us to ask: Is reasoning truly a consistent performance booster for vision-language models? Figure 1: The examples where more reasoning yields less accuracy. We select Vision-R1 ( Huang et al. 2025 ) as a representitive multimodal reasoning model and evaluate on established benchmarks. Given the ground truth (GT) answer, we examine three settings, i.e . , direct (DT) answering, full reasoning, and early decision. Correct and incorrect responses are highlighted in green and red , respectively, while designed prompts for early decision are indicated in blue . Motivated by this question, we perform a zoom-in analysis of how the reasoning process impacts accuracy of existing VLMs. Beyond the standard settings of direct answering and full-step reasoning, we introduce an early decision mode, where the model is prompted to halt reasoning at selected logical breakpoints and produce an immediate answer. This setup allows us to measure the contribution of each reasoning segment to the final outcome. Interestingly, our finding reveals that while early-stage reasoning may offer substantial gains, its advantages tend to saturate and may even reverse in later steps. To understand this phenomenon, we conduct an error analysis of the reasoning content in failure cases and find that perception errors, where the model incorrectly interprets visual details in the image, constitute the dominant category, accounting for over 50\% . Moreover, the majority of these errors can be rectified with a shorter reasoning via early decision, indicating that the model is initially capable of producing the correct answer but is ultimately misled by extended reasoning. For instance, as shown in Fig. 1 , the early stage of reasoning paths, i.e . , early decision, enables the model to arrive at the correct answer, whereas prolonged thoughts, i.e . , full reasoning, often lead to errors such as chart misreading or hallucination. This indicates the dual nature of reasoning: while it strengthens logical inference and facilitates complex problem-solving, it may gradually weaken perceptual grounding, leading to susceptibility on otherwise straightforward visual questions. This decline in perceptual capability constitutes a major bottleneck for multimodal reasoning. To further investigate its underlying causes, we examine how attention to visual tokens evolves throughout the reasoning process. Our analysis reveals a marked decrease in visual attention as reasoning progresses, suggesting the emergence of reasoning may inadvertently reduce the model’s reliance on visual input, a phenomenon we refer to as visual forgetting. To validate this hypothesis, we propose two interventions: 1) visual replay, which reintroduces the input image to models at regular intervals during reasoning; 2) focus prompt, which prompts models to attend to the visual input at selected steps. Both methods are proven to alleviate performance degradation induced by reasoning, providing evidence that visual forgetting is a central factor constraining reasoning potentials. While the two aforementioned remedies can partially alleviate visual forgetting, they incur substantial computational overhead during inference and fail to rectify the underlying behavioral deficiencies of existing reasoning models. To address this, we propose Vision-Anchored Policy Optimization ( VAPO ), a simple yet effective policy gradient algorithm that explicitly guides the reasoning process toward a visually grounded trajectory. The key idea is to embed a sequence of visual anchors throughout the reasoning path. At each anchor, the model’s perceptual capability is assessed by evaluating its responses to a set of primitive visual claims. Beyond standard outcome-based rewards such as accuracy and format, we introduce perception reward, which quantifies the model’s overall perceptual grounding during reasoning by aggregating scores across all anchor points. Experimental results demonstrate that our result model, VAPO-Thinker-7B , significantly enhances the model’s reliance on visual input during reasoning and achieves average gains of 2\sim 4\% over strong baselines on established benchmarks, setting a new state of the art. In summary, our contributions are as follows: • Despite the promise of multimodal reasoning, we identify a pronounced dual nature, where reasoning enhances logical inference capability at the expense of perceptual accuracy, constituting a major bottleneck to the overall effectiveness. • We further examine this side effect of reasoning and empirically demonstrate that it stems from visual forgetting, where prolonged reasoning gradually reduces the model’s reliance on visual input, leading to a substantial increase in perceptual failures. • We propose VAPO , a straightforward yet effective

Read the complete text