OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06

Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, Jun Wang · 2025

Read OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 by Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, Jun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16081.txt.

Introduction

Zhiyuan Liu and co-authors present OThink-MR1, a model that reads both images and text and is trained to reason more broadly, in a 2025 paper. The catalog title carries stray template text, but the paper itself is about this model. Most training of such models uses step-by-step teaching, which boosts set tasks but does little for broad reasoning. Learning by trial and reward looks more hopeful, yet it has weak points the team set out to fix.

Their fix changes how a key training limit shifts over time, rather than holding it fixed. On one base model, the new method beat plain teaching by more than five percent on same-task tests. It also beat the standard reward method by a wider margin. Gains were far larger when the trained model faced a new task, which the authors read as a sign of real, transferable reasoning skill.

Catalog record

CREATORZhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, Jun Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OThink-MR1: Stimulating multimodal generalized reasoning capabilities via dynamic reinforcement learning DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, Jun Wang Abstract . Multimodal Large Language Models (MLLMs) have gained significant traction for their ability to process diverse input data types and generate coherent, contextually relevant outputs across various applications. While supervised fine-tuning (SFT) has been the predominant approach to enhance MLLM capabilities in task-specific optimization, it often falls short in fostering crucial generalized reasoning abilities. Although reinforcement learning (RL) holds great promise in overcoming these limitations, it encounters two significant challenges: (1) its generalized capacities in multimodal tasks remain largely unexplored, and (2) its training constraints, including the constant Kullback-Leibler divergence or the clamp strategy, often result in suboptimal bottlenecks. To address these challenges, we propose OThink-MR1, an advanced MLLM equipped with profound comprehension and reasoning capabilities across multimodal tasks. Specifically, we introduce Group Relative Policy Optimization with a dynamic Kullback-Leibler strategy (GRPO-D), which markedly enhances reinforcement learning (RL) performance. For Qwen2-VL-2B-Instruct, GRPO-D achieves a relative improvement of more than 5.72% over SFT and more than 13.59% over GRPO in same-task evaluation on two adapted datasets. Furthermore, GRPO-D demonstrates remarkable cross-task generalization capabilities, with an average relative improvement of more than 61.63% over SFT in cross-task evaluation. These results highlight that the MLLM trained with GRPO-D on one multimodal task can be effectively transferred to another task, underscoring the superior generalized reasoning capabilities of our proposed OThink-MR1 model. 1. Introduction Multimodal Large Language Models ( Liang et al. 2024 ) integrate and process multiple types of input data, such as text and images, to generate coherent and contextually relevant outputs. These models aim to enhance multimodal understandingby leveraging the strengths of different modalities and have been widely used in applications such as Visual Question Answering ( Kuang et al. 2024 ; Lee et al. 2024 ) and Image Captioning ( Bucciarelli et al. 2024 ; Li et al. 2024 ) . Compared to unimodal large language models (LLMs), MLLMs must process more complex data structures, making their training process significantly more resource-intensive. Furthermore, the multimodal nature of MLLMs positions them as ideal candidates for deployment in complex and dynamic real-world scenarios. These factors, in turn, place higher demands on the models’ cross-task generalization capabilities–the ability to handle different tasks with different data distributions. Figure 1 . Test accuracy metric curves of SFT and GRPO on geometry reasoning task. Currently, most MLLMs rely on SFT ( Lin et al. 2024 ; Wang et al. 2024b ) as the primary post-training approach. However, research ( Chu et al. 2025 ) has shown SFT tends to memorize patterns in the training data, performing poorly in out-of-distribution (OOD) scenarios ( Chu et al. 2025 ) , not to mention cross-task generalization in multimodal settings. This limitation highlights the need for more robust training paradigms that can enhance the generalization capabilities of MLLMs. Recent advancements in LLMs, such as DeepSeek R1 ( Guo et al. 2025 ) , have demonstrated the effectiveness of reinforcement learning with verifiable rewards (RLVR) in improving reasoning abilities. These methods ( Team et al. 2025 ; Guo et al. 2025 ) leverage reinforcement fine-tuning to significantly enhance the reasoning performance of models. However, existing research has primarily focused on unimodal tasks, leaving the potential of RLVR for multimodal tasks and its ability to enable cross-task generalization largely unexplored . We directly extend reinforcement learning with verifiable rewards to enhance MLLMs for multimodal tasks. Specifically, we train and validate on the multimodal geometry reasoning task. The test accuracy metric curves of SFT and GRPO (Group Relative Policy Optimization ( Guo et al. 2025 ) , a method based on RLVR) are shown in Figure 1 . As the number of training steps increases, both SFT and GRPO exhibit performance improvements. However, SFT demonstrates a more significant enhancement and eventually surpasses GRPO when the training steps reach thousands. This can be attributed to GRPO’s insufficient exploration, leading to suboptimal solutions . To address these issues, we propose OThink-MR1, an advanced MLLM equipped with generalized understanding and reasoning across multimodal tasks. First, to address GRPO ’s insufficient exploration capacity, we propose a D ynamic KL strategy(GRPO-D) inspired by the \epsilon -greedy strategy in classical reinforcement learning. This strategy follows the principle of "early exploration, late exploitation" by dynamically adjusting the KL divergence to balance exploration and exploitation. The validation results demonstrate the dynamic KL strategy effectively enables GRPO-D to surpass SFT in same-task evaluation. Additionally, to assess the cross-task generalization ability, we propose cross-task validation, where models are post-trained on one task type and evaluated on another. We adopt the visual counting task (a multimodal content understanding task that extracts basic multimodal information) and the geometry reasoning task (a multimodal reasoning task that requires logical analysis of visual relationships). The examples of these two tasks are provided in Figure 2 . The cross-task validation results demonstrate the strong generalization capabilities of GRPO-D, highlighting its ability to effectively transfer knowledge across different tasks. To sum up, our contributions are listed below: • We propose OThink-MR1, a dynamic reinforcement learning framework for fine-tuning MLLMs, which outperforms SFT in the same-task validation. This approach dynamically balances exploration and exploitation, resulting in more effective learning. • We are among the first to demonstrate significant cross-task generalization of dynamic reinforcement learning for MLLMs. This demonstrates that models post-trained with GRPO-D on one multimodal task can be effectively transferred to other multimodal tasks, greatly reducing the need for extensive task-specific data collection and retraining across diverse applications. • We conduct extensive experiments on various multimodal tasks, including visual counting, geometry reasoning, cross-task validation and more, to verify our model’s generalized reasoning ability. Figure 2 . Examples of multimodal content understanding and multimodal reasoning tasks. Figure 3 . Qualitative illustration of GRPO-D vs SFT in OThink-MR1. 2. Related Work Multimodal Large Language Models such as LLaVA ( Liu et al. 2023 ) integrate and process multimodal inputs (e.g., text and images) to understand and generate coherent content. Training of MLLMs typically involves pre-training and post-training stages, where post-training is crucial for enhancing model performance. Most works ( Lu et al. 2022 ; Cheng et al. 2024 ; Zeng et al. [n. d.] ; Zhang et al. 2023 ) rely on SFT, while a few ( Liu et al. 2025 ; Chen et al. 2025 ) explore RL. However, SFT often struggles with memorization rather than generalization. Reinforcement learning (RL), particularly with verifiable rewards (e.g., GRPO-based RLVR in Visual-RFT ( Liu et al. 2025 ) ), excels in few-shot learning scenarios. However, as data scales up, RLVR may be outperformed by SFT due to imbalanced exploration and exploitation, leading to suboptimal solutions. Additionally, prior works, such as R1-V ( Chen et al. 2025 ) , have evaluated RLVR’s out-of-distribution (OOD) performance but are limited to same-task validation, where models are tested only on different data distributions within the same task domain. To address these issues, we propose a dynamic reinforcement learning algorithm, named GRPO-D. This algorithm adapts an "exploration early, exploitation late" strategy during training, enhancing

Read the complete text