AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in Unified Multimodal Models via Decompositional Verifiable Reward
Runhui Huang, Jie Wu, Rui Yang, Zhe Liu, Hengshuang Zhao · 2026
Read AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in Unified Multimodal Models via Decompositional Verifiable Reward by Runhui Huang, Jie Wu, Rui Yang, Zhe Liu, Hengshuang Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-12495.txt.
Introduction
Huang and four colleagues apply group relative policy optimisation to unified multimodal models. Their AlphaGRPO framework needs no extra cold-start stage. It unlocks two skills. First, text-to-image generation from reasoning, where the model infers hidden user intent. Second, self-reflective refinement, where it spots and fixes its own misalignments. A decompositional verifiable reward breaks each request into atomic, checkable questions. Those questions are scored by a general multimodal model. Tests show robust gains across generation benchmarks.
Catalog record
| CREATOR | Runhui Huang, Jie Wu, Rui Yang, Zhe Liu, Hengshuang Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67830 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.12495
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in Unified Multimodal Models via Decompositional Verifiable Reward Runhui Huang, Jie Wu, Rui Yang, Zhe Liu, Hengshuang Zhao Abstract In this paper, we propose AlphaGRPO, a novel framework that applies Group Relative Policy Optimization (GRPO) to AR-Diffusion Unified Multimodal Models (UMMs) to enhance multimodal generation capabilities without an additional cold-start stage. Our approach unlocks the model’s intrinsic potential to perform advanced reasoning tasks: Reasoning Text-to-Image Generation, where the model actively infers implicit user intents, and Self-Reflective Refinement, where it autonomously diagnoses and corrects misalignments in generated outputs. To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward ( DVReward ). Unlike holistic scalar rewards, DVReward utilizes an LLM to decompose complex user requests into atomic, verifiable semantic and quality questions, which are then evaluated by a general MLLM to provide reliable and interpretable feedback. Extensive experiments demonstrate that AlphaGRPO yields robust improvements across multimodal generation benchmarks, including GenEval, TIIF-Bench, DPG-Bench and WISE, while also achieving significant gains in editing tasks on GEdit without training on editing tasks. These results validate that our self-reflective reinforcement approach effectively leverages inherent understanding to guide high-fidelity generation. Project page: https://huangrh99.github.io/AlphaGRPO/ 1 Introduction Figure 1 : Qualitative and quantitative comparisons of AlphaGRPO. In Text-to-Image (top), our AlphaGRPO(trained on self-reflective refinement (SRR)) exhibits superior initial composition compared to Bagel, while applying Inference-time Self-Reflective Refinement (Inf. SRR) further rectifies fine-grained attribute mismatches (e.g., the “metallic” correct to “fabric” textures). In Image Editing (bottom), while the BAGEL baseline fails to capture the “cyberpunk” style, without training on the image editing task, AlphaGRPO trained on reasoning text-to-image (RT2I) generation demonstrates effective improvement on this task, and AlphaGRPO trained on SRR masters the style transformation. Quantitative comparison demonstrates that AlphaGRPO could consistently improve across five downstream benchmarks. Recent advancements in Unified Multimodal Models (UMMs) focus on designing unified architectures capable of seamlessly integrating visual understanding and generation ( 42 ; 45 ; 54 ; 44 ; 34 ; 17 ; 7 ; 53 ) , marking a distinct shift from pure AR to hybrid AR-Diffusion architectures. Distinct from specialized models, these unified models possess the innate capability to process interleaved multimodal inputs and outputs. Crucially, this structural unification endows them with the potential to orchestrate complex cognitive workflows within a single end-to-end model, encompassing reasoning, execution, self-reflection, and refinement. However, effectively reinforcing UMMs to leverage their intrinsic understanding to improve multimodal generation remains a largely unexplored challenge. Reinforcement Learning (RL), notably Group Relative Policy Optimization (GRPO) ( 41 ) , has demonstrated remarkable success in reinforcing reasoning capabilities in LLMs ( 41 ; 12 ) and optimizing visual generation in flow-matching diffusion models ( 29 ; 58 ) . To enable complex tasks like reasoning text-to-image generation or self-reflective refinement, recent works ( 7 ; 49 ; 18 ) primarily rely on proprietary models to synthesize high-quality data. Although effective, this paradigm inevitably introduces an additional cold-start SFT stage, implying that the performance gains might stem from the distillation of the stronger teacher models. In contrast, we argue that since unified models already acquire fundamental primitives and implicit reasoning-related data through large-scale pretraining, it is possible to activate and enhance these dormant capabilities using RL without the cold-start stage. The success of applying GRPO in multimodal generation relies on a reward model yielding stable, robust signals. To enhance broad, real-world multimodal generation capabilities, such a reward model is required to accurately assess diverse real-world samples. However, current visual generation RL often overlooks this, chasing high scores on training-aligned metrics ( 29 ; 58 ) . This risks reward overfitting and fails to guarantee consistent improvements across diverse downstream benchmarks. In the pursuit of a universal evaluator, Multimodal Large Language Models (MLLMs) have emerged as the premier candidates, due to their robust understanding capabilities and extensive world knowledge. Fine-tuning these models on human preference datasets can yield specialized reward models with improved alignment accuracy ( 46 ; 31 ; 55 ) . However, it shifts the model’s distribution towards a limited domain and implicitly narrows the MLLM’s capacity to handle open-world samples. Therefore, it becomes crucial to explore the stable, high-quality reward signals from general MLLMs without compromising their inherent understanding. In this paper, we propose AlphaGRPO, a novel framework that extends GRPO to multimodal generation in AR-Diffusion UMM. It enhances unified multimodal understanding and generation capabilities by unlocking the model’s intrinsic potential, without an additional cold-start stage. Specifically, we formulate multimodal generation as the unified trajectory that first generates text, then the image. We focus on the self-reflective refinement, which requires autonomously diagnosing misalignments from the initial generation results and executing correction strategies. This process demands a comprehensive synergy of capabilities, including multimodal perception, understanding, and generation. We introduce the False-Positive Rectification to eliminate the false improvement signals during training. Furthermore, we apply AlphaGRPO to reasoning text-to-image generation to validate the generalizability and robustness of AlphaGRPO across diverse multimodal tasks. To ensure reliable reward signals and promote robustness in real-world scenarios, we introduce the Decompositional Verifiable Reward (DVReward). This mechanism utilizes an LLM to decompose complex user requests into atomic, verifiable questions and verify them against the generated visual content using MLLM confidence scores. In our experiments, we prioritize evaluating the method’s generalization ability across diverse downstream tasks, rather than relying on in-distribution test sets. As illustrated in Figure 1 , powered by DVReward, AlphaGRPO training on both reasoning T2I (RT2I) and self-reflective refinement consistently improves performance on image generation and image editing benchmarks. Furthermore, in the Self-Reflective Refinement task, without training on editing data, AlphaGRPO not only maintains comparable gains to AlphaGRPO (RT2I) on image generation benchmarks and secures a 0.52 improvement on editing benchmark, i.e., GEdit ( 30 ) , validating generalizability. Moreover, leveraging the inference-time self-reflective refinement further elevates T2I performance, reaching 83.9% on TIIF-Bench and outperforming Bagel by 5.8%. The contributions of this paper can be summarized: • We propose AlphaGRPO, the first framework to introduce GRPO training to AR-Diffusion Unified Models. By eliciting the model’s latent primitives without an additional cold-start stage, we enable advanced capabilities in both Reasoning Text-to-Image Generation and Self-Reflective Refinement. • We introduce Decompositional Verifiable Reward (DVReward), a novel fine-grained reward mechanism that decomposes user prompts into atomic verifiable questions across both semantic alignment and visual fidelity. This approach provides stable, interpretable supervision signals for multimodal generation GRPO training that indicate the correct way to use MLLM as the reward model. • Our experiments demonstrate that AlphaGRPO achieves consistent and significant improvements across multimodal generation benchmarks (e.g., GenEval, TIIF-Bench) and multimodal editing tasks (e.g., GEdit), proving the effectiveness and generalizability of AlphaGRPO. 2 Pilot study Before detailing our methodology, we conducted a pilot study to investigate two fundamental premises essential for aligning unified Multimodal Large Language Models (MLLMs): (1) whether pretrained UMMs possess the latent reasoning patterns required for self-reflective refinement and how to activate this, and (2) whether current MLLMs can provide reliable, discriminative reward signals to evaluate visual generation in open-world scenarios. Explicit error-seeking activates latent reasoning. To explore how to activate latent reasoning, we probe the state-of-the-art UMM, Bagel, with two tasks: Verification, where the model judges whether misalignments exist between the generated image and