Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models
Jiajun Fan, Tong Wei, Chaoran Cheng, Yuxin Chen, Ge Liu · 2025
Read Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models by Jiajun Fan, Tong Wei, Chaoran Cheng, Yuxin Chen, Ge Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-18053.txt.
Introduction
Jiajun Fan, with four co-authors, makes the safety leash on fine-tuning generative models adjust itself. A fixed leash creates a bind: tight control guards skill but caps reward, while loose control invites cheating and collapse. Their ADRPO method loosens the leash for high-value samples and tightens it for poor ones.
Strength is set from advantage estimates, so the model explores hard where the payoff looks real. Applied with a distance-based leash to flow matching models, the method improved text-to-image fit in their tests. Trainers of image models get a tuning rule that follows data quality instead of a fixed schedule.
Catalog record
| CREATOR | Jiajun Fan, Tong Wei, Chaoran Cheng, Yuxin Chen, Ge Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-18053
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Divergence Regularized Policy Optimization for Fine-tuning Generative Models Jiajun Fan, Tong Wei, Chaoran Cheng, Yuxin Chen, Ge Liu Abstract Balancing exploration and exploitation during reinforcement learning fine-tuning of generative models presents a critical challenge, as existing approaches rely on fixed divergence regularization that creates an inherent dilemma: strong regularization preserves model capabilities but limits reward optimization, while weak regularization enables greater alignment but risks instability or reward hacking. We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), which automatically adjusts regularization strength based on advantage estimates—reducing regularization for high-value samples while applying stronger regularization to poor samples, enabling policies to navigate between exploration and aggressive exploitation according to data quality. Our implementation with Wasserstein-2 regularization for flow matching generative models achieves remarkable results on text-to-image generation, achieving better semantic alignment and diversity than offline methods like DPO and online methods with fixed regularization like ORW-CFM-W2. ADRPO enables a 2B parameter SD3 model to surpass much larger models with 4.8B and 12B parameters in attribute binding, semantic consistency, artistic style transfer, and compositional control while maintaining generation diversity. ADRPO generalizes to KL-regularized fine-tuning of both text-only LLMs and multi-modal reasoning models, enhancing existing online RL methods like GRPO. In LLM fine-tuning, ADRPO demonstrates an emergent ability to escape local optima through active exploration, while in multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning, enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, offering an effective plug-and-play solution to the exploration-exploitation challenge across diverse generative architectures and modalities. 1 Introduction Reinforcement learning fine-tuning has emerged as a powerful paradigm for aligning generative models with human preferences, driving remarkable improvements in capabilities from text generation to image synthesis ( instructed_rlhf ; DRLHF2017 ; stiennon2020 ) . At the core of modern RLHF approaches lies a fundamental challenge: effectively balancing divergence regularization against reward maximization during policy optimization. This balance is critical as it determines whether models retain the beneficial properties of their pre-trained foundation while adapting to better satisfy human preferences ( dpo ; zie2020 ; user_def1 ) . The current standard practice employs divergence regularization with fixed coefficients to constrain policy updates - typically using Kullback-Leibler (KL) ( grpo ; instructed_rlhf ) or Wasserstein-2 (W2) divergences ( wgan ; ORW-CFM-W2 ) . However, this approach creates an inherent dilemma that limits performance: strong regularization preserves model capabilities but hampers reward optimization, while weak regularization enables greater reward optimization but risks catastrophic forgetting, mode collapse, or reward hacking ( forget ; ai_collapse_nature ) . This trade-off is particularly pronounced in generative models where preserving diversity while improving quality represents a critical balance ( ddpo ; dpok ) . Existing approaches like PPO ( ppo ) , GRPO ( grpo ) , and DPO ( dpo ; diff_dpo ) employ fixed regularization coefficients that treat all data points equally, regardless of whether the policy should prioritize exploitation (when rewards are reliable) or exploration (when falling into suboptimal solutions). This one-size-fits-all approach fails to adapt to the varying exploration needs across the complex landscapes of generative model policy optimization. To address these limitations, we propose Adaptive Divergence Regularized Policy Optimization (ADRPO), a novel framework that dynamically adjusts regularization strength using advantage estimates ( gae ; rlsutton ) . ADRPO employs advantage signals to fine-tune the exploration-exploitation trade-off: high-advantage samples reduce regularization for aggressive optimization, while low-advantage samples increase it for stability. This sample-level adaptation integrates seamlessly into training, providing an efficient and automated approach. By aligning regularization with sample quality, ADRPO overcomes the shortcomings of prior methods, delivering superior alignment and generative performance across diverse tasks, as demonstrated in our experiments with text-to-image alignment and language model fine-tuning. In summary, our approach makes several important contributions: 1. General RL Framework with Adaptive Divergence Regularization. We introduce ADRPO as a general-purpose framework that dynamically adjusts regularization based on advantage estimates, integrating with existing RL fine-tuning methods without architectural changes. Our proposed methods address the exploration-exploitation dilemma while preventing reward hacking and model collapse. 2. Superior Text-to-Image Alignment with Smaller Model. We first propose a novel online RL method based on ADRPO, combining advantage-based policy optimization and adaptive W2 regularization for fine-tuning flow matching models. Our experiments of fine-tuning SD3 demonstrate ADRPO’s dominant Pareto frontier in the reward-diversity trade-off and reward-divergence trade-off compared to DPO ( dpo ) and fixed-regularization approaches ( ORW-CFM-W2 ) (See Figs. 2 and 3 ). Notably, our 2B parameter model outperforms larger 4.8B ( sana15 ) and 12B ( fluxdev ) parameter models across attribute binding, compositional control, and semantic consistency (See Figs. 1 and Tab. 1 ). 3. Emergent Exploration in LLMs. We also apply our ADRPO to improve GRPO ( grpo ) for online fine-tuning of LLMs (See Figs. 4 ). ADRPO not only improves alignment but exhibits an emergent ability to escape local optima by actively increasing exploration when needed—a capability absent in fixed-regularization methods like GRPO. 4. Cross-Domain Applicability. ADRPO provides a unified solution across continuous (flow matching with W2 regularization), discrete (LLMs with KL divergence), and multi-modal reasoning generative paradigms (audio reasoning models), offering immediate practical benefits with minimal computational overhead (See Sec. 4.6 and App. D.3 ). 2 Related Work RL Fine-tuning for LLMs. Reinforcement learning has become the dominant approach for aligning large language models with human preferences. Pioneering work by ( DRLHF2017 ) established the RLHF framework, which was later scaled by ( instructed_rlhf ) to create models that better follow human instructions. The algorithmic landscape has evolved from PPO ( ppo ) to more efficient alternatives like GRPO ( grpo ) and offline approaches like DPO ( dpo ) . These methods have significantly improved the reasoning capabilities of models like DeepSeek-R1 ( guo2025deepseek_r1 ) and improved their instruction-following abilities. Despite their success, these approaches typically rely on fixed regularization parameters that treat all samples equally, regardless of whether they represent promising directions for optimization or uncertainty-laden regions requiring more conservative updates (See Fig. 4 ). RL Fine-tuning for Flow Matching Models. While RL fine-tuning is established for language models, its application to flow matching (FM) models ( fmgm ) presents unique exploration-exploitation trade-off challenges due to their continuous-time nature and ODE-based sampling. Recent approaches like Online Reward-Weighted Fine-Tuning with Wasserstein regularization ( ORW-CFM-W2 ) and offline methods like diffusion-DPO ( diff_dpo ) have made progress, but remain limited by fixed regularization schemes that cannot adapt to sample-specific characteristics. This fundamental limitation restricts their ability to optimally balance the critical exploration-exploitation trade-off necessary for effective fine-tuning of state-of-the-art image generation models like SD3 ( sd3 ) (See Tab. 1 and Figs. 2 ). Divergence Regularization in RL Fine-tuning. Divergence regularization plays a crucial role in RL fine-tuning by preventing the policy from deviating too far from the initial model, thus preserving desirable properties while allowing for improvement. For language models, KL divergence serves as the standard metric in methods like PPO ( ppo ) , GRPO ( grpo ) , and DPO ( dpo ) , while flow models benefit from Wasserstein distances ( ORW-CFM-W2 ) that better handle continuous distributions. Despite their importance, existing approaches typically employ fixed regularization coefficients that fail to handle the varying significance of regularization across different samples