VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning
Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen · 2025
Read VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning by Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08837.txt.
Introduction
Wang and co-authors train vision-language models to slow down and rethink, using reinforcement learning alone. Text models that reflect openly have leapt ahead on math and science tests. Their multimodal cousins, though, still match fast models on visual math benchmarks. The team adapts the GRPO training method for vision models. A trick called Selective Sample Replay fixes a fading-signal problem in training. Early runs gained strong scores with the fix.
Yet the trained models showed little true self-checking. They rarely paused to verify or revise an answer. That gap between score and habit is the paper's focus. The authors push training rewards toward genuine reflection, not just right answers.
Catalog record
| CREATOR | Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08837
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, Wenhu Chen Abstract Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated great potential in solving challenging problems through explicit reflection. They significantly outperform the best fast-thinking models, such as GPT-4o, on various math and science benchmarks. However, their multimodal reasoning capabilities remain on par with fast-thinking models. For instance, GPT-o1’s performance on benchmarks like MathVista, MathVerse, and MathVision is similar to fast-thinking models. In this paper, we aim to enhance the slow-thinking capabilities of vision-language models using reinforcement learning (without relying on distillation) to advance the state of the art. First, we adapt the GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to address the vanishing advantages problem. While this approach yields strong performance, the resulting RL-trained models exhibit limited self-reflection or self-verification. To further encourage slow-thinking, we introduce Forced Rethinking, which appends a rethinking trigger token to the end of rollouts in RL training, explicitly enforcing a self-reflection reasoning step. By combining these two techniques, our model, VL-Rethinker , advances state-of-the-art scores on MathVista, MathVerse to achieve 80.4%, 63.5% respectively. VL-Rethinker also achieves open-source SoTA on multi-disciplinary benchmarks such as MathVision, MMMU-Pro, EMMA, and MEGA-Bench, narrowing the gap with OpenAI-o1. Our empirical results show the effectiveness of our approaches. Figure 1 : Performance comparison between VL-Rethinker and other SoTA models on different multimodal reasoning benchmarks. 1 Introduction Recently, slow-thinking systems such as OpenAI-o1 ( Jaech et al. 2024 ) , DeepSeek-R1 ( Guo et al. 2025 ) , Kimi-1.5 ( Team et al. 2025 ) , Gemini-Thinking ( Team et al. 2023 ) , and QwQ/QvQ ( Bai et al. 2025 ) have significantly advanced the performance of language models in solving challenging math and science problems. These models engage in extended reasoning and reflection before arriving at a final answer, in contrast to fast-thinking models like GPT-4o ( Hurst et al. 2024 ) and Claude-3.5-Sonnet ( Anthropic 2024 ) , which produce answers rapidly without such deliberation. Through this reflective process, slow-thinking models outperform the best fast-thinking models by over 30% on math datasets such as AIME24 and AMC23 ( Hendrycks et al. ) , and by around 10% on general science benchmarks like GPQA ( Rein et al. 2024 ) . However, their multimodal reasoning capabilities remain on par with fast-thinking models. For example, GPT-o1 achieves 73.9% on MathVista ( Lu et al. 2023 ) and 57.0% on MathVerse ( Wang et al. 2024a ) , which is slightly worse than Qwen2.5-VL-72B ( Wang et al. 2024b ) scoring 74.8% and 57.2% on the same benchmarks. This raises an important research question: How can we effectively incentivize multimodal slow-thinking capabilities in Vision-Language Models? To address this, we explore how to directly train multimodal reasoning models through reinforcement learning (RL), without relying on distillation from stronger teacher models ( Yang et al. 2025 ; Deng et al. 2025 ) . Our main contributions are as follows: GRPO with SSR: We construct a dataset of 38,870 queries covering a diverse range of topics for training our vision-language model (VLM). We adapt the Group Relative Policy Optimization (GRPO) algorithm ( Guo et al. 2025 ) , which computes advantages by comparing responses within the same query group and normalizes rewards to guide policy updates. However, we identify a key challenge with GRPO: the vanishing advantages problem. This occurs when all responses in a group receive identical rewards (either all correct or all incorrect), leading to zero advantage signals and ineffective gradient updates. This reward uniformity exacerbates instability as training progresses, hindering the model from exploring deeper reasoning. To mitigate this, we introduce Selective Sample Replay (SSR), which enhances GRPO by integrating an experience replay mechanism that samples high-value experiences from past iterations. SSR augments the current training batch with rehearsed samples that previously indicated large magnitudes of advantages. This strategic experience replay counteracts the Vanishing Advantages problem and provides more consistent gradient signals. Furthermore, SSR embodies the principles of curriculum learning ( Team et al. 2025 ) in an online and active fashion Lightman et al. 2023 , by dynamically adjusting the training focus towards high-value experiences situated near the model’s decision boundaries. While this approach demonstrates strong empirical performance across several multimodal reasoning benchmarks, we observe that the resulting models still exhibit limitations in explicit reflective behavior, suggesting avenues for further improvement. Forced Rethinking: To address this, we propose a simple yet effective technique called forced rethinking. We append a textual rethinking trigger to the end of roll-out responses and train the model using the same RL setup. This strategy prompts the model to engage in self-reflection and self-verification before producing the final answer. We name the resulting model VL-Rethinker . As shown in Fig. 1 , VL-Rethinker significantly outperforms GPT-o1 on mathematical benchmarks such as MathVista, MathVerse. Furthermore, on general-purpose multimodal benchmarks like EMMA and MMMU-Pro, VL-Rethinker achieves a new open-source state of the art performance, closely approaching GPT-o1’s performance. Observations: We observe a notable discrepancy between modalities: while RL training often induces slow-thinking behaviors such as longer reasoning traces in math-focused tasks ( Zeng et al. 2025 ; Wen et al. 2025 ) , vision-language tasks rarely exhibit such development. Specifically, models trained on multimodal data do not naturally adopt longer chains of thought or spontaneous wait patterns. Understanding why RL incentivizes reflection differently in multimodal contexts versus math-only settings is an important avenue for future work. In summary, our contributions are threefold: (1) We propose and validate a simple, direct RL approach for enhancing VLM reasoning, offering a viable alternative to complex supervised fine-tuning and distillation pipelines. (2) We introduce Selective Sample Replay (SSR) to improve the training stability and effectiveness of GRPO-based RL for VLMs. (3) We propose Forced Rethinking, a lightweight yet powerful strategy to incentivize self-reflection in VLMs. Our final model, VL-Rethinker , sets a new state of the art on key multimodal reasoning benchmarks, demonstrating the value of slow-thinking reinforcement in vision-language modeling. 2 Preliminaries This section outlines the key concepts and training setup for multimodal reasoning. We first formulate the multimodal reasoning problem and define our learning objective. Then, we describe the standard Reinforcement Learning (RL) algorithm used in our framework. 2.1 Problem Formulation We define the multimodal reasoning task as follows: given a multimodal input consisting of one or more images I and a textual query Q , the goal is to generate a textual response y that correctly answers the query by reasoning over both visual and textual information. Let \mathcal{V} denote the visual input space and \mathcal{T} the textual input space. The input is denoted as x\in\mathcal{V}\times\mathcal{T} , where x=(I,Q) captures both modalities. The output is a textual response y\in\mathcal{Y} , where \mathcal{Y} represents the response space. The challenge lies in building a vision-language model (VLM) that can integrate multimodal information and perform deep, multi-step reasoning—especially for complex queries requiring extended deliberation or external knowledge. Our goal is to improve the reasoning capabilities of an instruction-tuned VLM that initially exhibits fast-thinking behavior, i.e., producing shallow, immediate responses. We aim to shift the model toward slow-thinking behavior—engaging in deeper, more deliberate reasoning—to significantly improve performance on