Co-Reinforcement Learning for Unified Multimodal Understanding and Generation

Jingjing Jiang, Chongjie Si, Jun Luo, Hanwang Zhang, Chao Ma · 2025

Read Co-Reinforcement Learning for Unified Multimodal Understanding and Generation by Jingjing Jiang, Chongjie Si, Jun Luo, Hanwang Zhang, Chao Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17534.txt.

Introduction

Jingjing Jiang and co-authors train one AI model to both understand and create images and text, using reinforcement learning for the two skills together. Their 2025 paper studies unified multimodal models, which usually learn each skill apart. Pilot tests suggested the two skills can grow together inside one shared training loop, each feeding the other.

Their framework, CoRL, runs a joint stage that rewards both skills, then a refined stage that polishes each task. The resulting model gained about seven percent on three image generation sets and twenty-three percent across nine understanding benchmarks. The authors argue that co-training beats treating the skills as separate trades.

Catalog record

CREATORJingjing Jiang, Chongjie Si, Jun Luo, Hanwang Zhang, Chao Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17534

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Co-Reinforcement Learning for Unified Multimodal Understanding and Generation Jingjing Jiang, Chongjie Si, Jun Luo, Hanwang Zhang, Chao Ma Abstract This paper presents a pioneering exploration of reinforcement learning (RL) via group relative policy optimization for unified multimodal large language models (ULMs), aimed at simultaneously reinforcing generation and understanding capabilities. Through systematic pilot studies, we uncover the significant potential of ULMs to enable the synergistic co-evolution of dual capabilities within a shared policy optimization framework. Building on this insight, we introduce CoRL , a Co - R einforcement L earning framework comprising a unified RL stage for joint optimization and a refined RL stage for task-specific enhancement. With the proposed CoRL, our resulting model, ULM-R1 , achieves average improvements of 7% on three text-to-image generation datasets and 23% on nine multimodal understanding benchmarks. These results demonstrate the effectiveness of CoRL and highlight the substantial benefits of reinforcement learning in facilitating cross-task synergy and optimization for ULMs. Code is available at https://github.com/mm-vl/ULM-R1 . 1 Introduction As large foundation models (LFMs) continue to advance in their general capabilities and breadth of knowledge, post-training [ 30 , 46 , 73 , 70 , 108 ] has emerged as a critical paradigm for further refining pretrained LFMs toward specialized applications, thereby facilitating task adaptation and human-aligned behaviors. Recently, reinforcement learning (RL)-based approaches [ 51 , 52 , 72 , 95 , 69 , 59 , 64 ] have exhibited considerable promise due to their data efficiency and strong alignment abilities. A notable exemplar is DeepSeek-R1 [ 19 ] , which demonstrates that RL with verifiable rewards and the group relative policy optimization (GRPO) algorithm constitutes a practical and stable strategy that sidesteps explicit preference modeling [ 77 ] and reward model learning [ 83 ] . This promising paradigm indicates the significant potential of LFMs to acquire advanced capabilities and generalize effectively without dependence on large-scale, high-quality supervised data . In the multimodal AI research community, the prevailing implementation [ 40 , 24 , 9 , 53 , 41 , 101 , 100 , 69 ] of the GRPO algorithm centers on crafting diverse rule-based reward mechanisms to incentivize long-chain reasoning capabilities of multimodal large language models (MLLMs). These initiatives primarily target multimodal understanding, with a particular focus on visual and mathematical reasoning tasks. Conversely, its application to visual generation remains surprisingly limited, with only pioneering explorations [ 79 , 25 ] suggesting its feasibility. More importantly, extending GRPO to unified MLLMs (ULMs) [ 8 , 90 , 88 , 39 ] capable of concurrently performing visual understanding and generation tasks remains considerably under-explored. Intuitively, ULMs could significantly benefit from GRPO owing to their inherent advantages of cross-task synergy and LLM sharing , which enables ULMs to share reward signals across various tasks and effectively mitigate reward imbalance, particularly as GRPO operates by jointly ranking outputs within task-agnostic groups. This paper aims to enhance the understanding and generation capabilities of ULMs without relying on supervised data. We begin with a set of pilot experiments to explore efficient reinforcement learning paradigms. Specifically, we systematically examine four rule-based training strategies: ( i ) separate RL for individual tasks, ( ii ) separate RL with weight merging, ( iii ) cycle RL alternating between tasks, and ( iv ) unified RL with joint optimization. Our explorations reveal two critical findings. First , direct task-specific RL fails to achieve the anticipated improvements, particularly in visual generation, and even impairs other abilities. Second , compared with alternative strategies, unified RL showcases comprehensive advantages across tasks. These results demonstrate the synergistic co-evolution of dual capabilities under a shared policy optimization paradigm. In light of our preliminary findings, we propose CoRL , a co-reinforcement learning framework designed to synergistically improve the understanding and generation capabilities of ULMs. Specifically, CoRL follows a Foundation-then-Specialization paradigm and is implemented through a two-stage RL procedure: a unified RL stage for joint optimization of dual capabilities and a refined RL stage for task-specific enhancement. In the first stage, the policy ULM is optimized through a unified GRPO algorithm with diverse rewards on a carefully curated dataset spanning both understanding and generation tasks. To effectively guide policy optimization in visual generation, we introduce a bidirectional cycle consistency reward and a text-image matching reward , which together promote semantic consistency and faithfulness between synthesized images and their corresponding prompts. The designed rewards complement typical multimodal understanding rewards ( i.e. , accuracy and format) within a unified group, enabling cross-task joint optimization. In the subsequent stage, we independently reinforce the policy’s understanding and generation capabilities using respective rewards and tailored datasets for task-specific refinement. Applying the two-stage CoRL procedure to the baseline ULM Janus-Pro [ 8 ] yields ULM-R1 , a unified model with reinforced capabilities in both understanding and generation. To comprehensively assess its performance, we conduct extensive comparisons against state-of-the-art unified MLLMs and dedicated models across both three visual generation and nine multimodal understanding benchmarks. Notably, ULM-R1 achieves substantial gains over its baseline on complex mathematical and logical reasoning tasks, such as WeMath (+15.2) and LogicVista (+10.6). These results underscore the effectiveness of CoRL, providing compelling empirical evidence for the efficacy of RL in jointly advancing visual understanding and generation tasks. We summarize our main contributions as follows: • We establish that RL with verifiable rewards and GRPO constitutes a data-efficient paradigm for cross-task co-optimization and capability enhancement. • We introduce a co-reinforcement learning framework, CoRL, to synergistically enhance the dual capabilities of ULMs using a unified-then-refined RL paradigm. • We demonstrate the effectiveness of CoRL and the advantage of ULM-R1 through extensive qualitative and quantitative experiments across diverse benchmarks. 2 Related Work Unified Multimodal Understanding and Generation. Recent advancements [ 87 , 8 , 39 , 88 , 86 , 71 , 90 , 107 , 91 , 82 , 67 , 76 ] have witnessed increasing attention to jointly model multimodal understanding and visual generation within a unified model. Pioneering attempts [ 17 , 12 ] predominantly rely on continuous diffusion models, integrating external diffusion decoders for image synthesis. Inspired by autoregressive next-token prediction, a growing line of research [ 71 , 87 , 82 , 85 , 8 , 57 , 39 , 88 , 86 , 31 , 47 , 111 ] encode visual inputs into discrete tokens and generate images in a fully autoregressive (F-AR) manner. Specifically, this approach employs a vector quantized (VQ) tokenizer [ 15 , 93 ] to convert images into discrete tokens, analogous to text tokenization. To mitigate information loss in VQ discretization, another stream of work [ 90 , 76 , 23 , 29 , 16 , 48 , 92 , 4 , 74 , 65 ] explores autoregressive and diffusion (AR-Diff) hybrid modeling approaches. Architecturally, these models typically comprise a vision autoencoder, a text tokenizer, and an LLM. Given the unified advantage of the F-AR model in generation manner, this work builds upon it to develop our co-reinforcement learning framework. RL-based Post-Training for MLLMs. Post-training [ 108 ] aims to further enhance the performance of pretrained models for customized applications and user needs. Recently, RL [ 84 , 78 ] has emerged as a powerful post-training technique, enabling models to learn from feedback and align with human

Read the complete text