Inpainting-Guided Policy Optimization for Diffusion Large Language Models
Siyan Zhao, Mengchen Liu, Jing Huang, Miao Liu, Chenyu Wang, Bo Liu, Yuandong Tian, Guan Pang, Sean Bell, Aditya Grover, Feiyu Chen · 2025
Read Inpainting-Guided Policy Optimization for Diffusion Large Language Models by Siyan Zhao, Mengchen Liu, Jing Huang, Miao Liu, Chenyu Wang, Bo Liu, Yuandong Tian, Guan Pang, Sean Bell, Aditya Grover, Feiyu Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10396.txt.
Introduction
Teaching language models by inpainting is Siyan Zhao and ten co-authors' idea in this 2025 paper. Masked diffusion models generate text by filling blanks. Standard models write left to right instead. Reinforcement learning for any such model suffers in a common way. Rewards are rare. Failed tries are wasted.
Their framework, IGPO, turns inpainting into a guide. During training, it inserts fragments of correct reasoning into the model's own attempts. The model still writes most of the solution itself. The fragments steer exploration toward promising paths. That rescues training signals. Group-based methods would otherwise zero those signals out. Fine-tuning on rewritten, concise traces adds a second boost. Entropy-based filtering adds a third. Together, the recipe sets new best results for its model class. The wins come on three math benchmarks, including GSM8K.
Catalog record
| CREATOR | Siyan Zhao, Mengchen Liu, Jing Huang, Miao Liu, Chenyu Wang, Bo Liu, Yuandong Tian, Guan Pang, Sean Bell, Aditya Grover, Feiyu Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-10396
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inpainting-Guided Policy Optimization for Diffusion Large Language Models Siyan Zhao, Mengchen Liu, Jing Huang, Miao Liu, Chenyu Wang, Bo Liu, Yuandong Tian, Guan Pang, Sean Bell, Aditya Grover, Feiyu Chen Abstract Masked diffusion large language models (dLLMs) are emerging as promising alternatives to autoregressive LLMs, offering competitive performance while supporting unique generation capabilities such as inpainting . We explore how inpainting can inform RL algorithm design for dLLMs. Aligning LLMs with reinforcement learning faces an exploration challenge: sparse reward signals and sample waste when models fail to discover correct solutions. While this inefficiency affects LLMs broadly, dLLMs offer a distinctive opportunity—their inpainting ability can guide exploration. We introduce IGPO (Inpainting Guided Policy Optimization), an RL framework that strategically inserts partial ground-truth reasoning traces during online sampling. Unlike providing full solutions, inpainting steers exploration toward promising trajectory spaces while preserving self-generated reasoning, bridging supervised fine-tuning and reinforcement learning. We apply IGPO to group-based optimization methods such as GRPO, where exploration failures cause zero advantages and gradients. IGPO restores meaningful gradients while improving sample efficiency. We also propose supervised fine-tuning on synthetically rewritten concise traces that better align with dLLM generation patterns. With additional techniques including entropy-based filtering, our training recipe yields substantial gains across three mathematical benchmarks—GSM8K, Math500, and AMC—achieving new state-of-the-art results for full-attention masked dLLMs. † † affiliation: Meta Superintelligence Labs † † affiliation: UCLA † † affiliation: Tsinghua University, College of AI † † affiliation: MIT † † contribution: Work done at Meta † † contribution: Core Contribution † † date: August 10, 2026 † † correspondence: Siyan Zhao at [email protected] , Feiyu Chen at [email protected] Figure 1: (a) Unlike autoregressive LLMs, diffusion LLMs can be conditioned on future reasoning hints during generation through inpainting via bidirectional attention, enabling guided exploration toward correct solutions. (b) Applying inpainting-guided exploration in policy optimization outperforms standard Group-relative Policy Optimization (GRPO) sampling and reduces all-wrong groups occurrences. (c) Our full training recipe combining Length-Aligned supervised fine-tuning on concise reasoning traces with IGPO achieves SoTA performance among full-attention masked dLLMs across three mathematical benchmarks. 1 Introduction Recent research has shown that masked diffusion large language models (dLLMs) ( Austin et al. 2021 ; Lou et al. 2024 ; Shi et al. 2024 ) such as LLaDA ( Nie et al. 2025 ) and Dream ( Ye et al. 2025 ) can achieve performance competitive with autoregressive models of similar size. Moreover, their capabilities and performance can be further enhanced via RL post-training ( Zhao et al. 2025 ) and ability to flexibly include multimodal data ( Li et al. 2025 ; Yang et al. 2025 ; You et al. 2025 ) . Unlike autoregressive models, which decode in a left-to-right manner, dLLMs iteratively unmask tokens in parallel. This formulation brings potential for faster inference as shown in closed models like Mercury ( Inception Labs et al. 2025 ) and Gemini Diffusion ( DeepMind 2025 ) , along with a flexible inductive bias for operations such as inpainting , the ability to fill in missing content within existing text. In this work, we explore how inpainting can be leveraged to inform post-training algorithms for dLLMs. Recent work on post-training alignment of dLLMs has adopted training approaches similar to autoregressive LLMs, applying Reinforcement Learning with Verifiable Reward (RLVR) methods and demonstrating promising results across reasoning tasks ( Zhao et al. 2025 ; Yang et al. 2025 ; Gong et al. 2025b ) . However, a fundamental exploration challenge persists: for challenging tasks, policies struggle to discover correct solutions and binary rewards provide minimal learning signal when most generated solutions are incorrect. This leads to substantial sample waste and poor training efficiency, exacerbating the computational costs of online RL. The bidirectional generative structure of diffusion models provides a unique mechanism to address this exploration challenge. Since dLLMs are trained through stochastic masking patterns, they possess inherent capability for accepting externally provided partial hints through inpainting operations. We leverage this ability to introduce \inpaint (Inpainting Guided Policy Optimization), a novel RL framework that strategically guides exploration for dLLMs by injecting reasoning hints when answering difficult problems. Specifically, when the policy is unlikely to generate correct solutions, partial reasoning traces are injected into the generation region, and the dLLM is tasked with completing the remaining reasoning sequence and output final answer. The final answers are verified against ground truth, and only successful completions are used for downstream policy optimization to mitigate the non-positive learning signal problem. We demonstrate \inpaint ’s effectiveness by applying it to group-based policy optimization methods such as Group-relative Policy Optimization (GRPO) ( Shao et al. 2024 ) . These methods are particularly vulnerable to exploration failures: when all sampled responses within a group produce uniformly incorrect (or correct) rewards, advantage estimation relies on group-relative reward normalization, causing the advantage signal to collapse to zero and resulting in zero gradients. This phenomenon occurs with alarming frequency in challenging reasoning domains, making group-based RL methods an ideal testbed for our approach. By reducing the prevalence of all-wrong groups, \inpaint restores non-degenerate gradient signals, accelerates convergence and enables more effective RL. More broadly, \inpaint can be viewed as a form of guided exploration that interpolates between supervised and reinforcement learning paradigms. The injected tokens function as conditioning context that steers the policy’s action distribution toward high-reward regions . Unlike pure SFT, which suffers from distribution shift between data and policy rollouts ( Zhang et al. 2025 ) , \inpaint maintains on-policy generation for the non-injected tokens, ensuring that gradient updates remain closer to the current policy’s actual sampling distribution. Finally, we augment \inpaint with additional reinforcement learning techniques that improve learning stability and performance, including entropy-based gradient filtering for inpainted tokens, and conduct comprehensive experiments across mathematical reasoning benchmarks. Through careful ablation analysis, we systematically evaluate each component of our approach to understand the mechanisms underlying inpainting-guided policy optimization. Our work makes the following key novel contributions: [leftmargin=.18in] • We propose IGPO, the first work to utilize the unique inpainting capabilities of diffusion LLMs for reinforcement learning. By strategically injecting partial reasoning traces during exploration, IGPO alleviates the inefficiency of sparse verifiable rewards and effectively mitigates the zero-advantage dilemma in group-based policy optimization methods such as GRPO. By substantially reducing the proportion of all-wrong groups (by \sim 60% as shown in Fig 1 (b)) and preserving output diversity, our approach yields non-degenerate gradients. • We propose a Length-Aligned supervised fine-tuning strategy for full-attention based dLLMs using synthetically rewritten, concise reasoning traces. This design better aligns SFT data length with RL sampling and evaluation length, avoids the limitations of verbose traces, and provides stronger initialization for downstream RL. • Our full training recipe achieves substantial improvements on mathematical reasoning benchmarks, including +4.9% on GSM8K, +8.4% on Math500, and +9.9% on AMC relative to the LLaDA-Instruct, achieving state-of-the-art performance among full-attention based dLLMs on these mathematical benchmarks. • We conduct a comprehensive ablation study that disentangles the mechanisms of IGPO. We show that partial inpainting consistently outperforms full ground-truth inpainting by staying closer to the policy distribution in online RL, and propose an entropy-based gradient filtering mechanism that stabilizes training dynamics. 2 Preliminaries 2.1 Masked Diffusion Large Language Models Masked diffusion LLMs ( Austin et al. 2021 ; Sahoo