Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models?

Renye Yan, Jikang Cheng, Shikun Sun, Yi Sun, You Wu, Wei Peng, Zongwei Wang, Ling Liang, Junliang Xing, Yimao Cai, [0.2cm] Peking University, Tsinghua University, Nanjing University, Stanford University · 2026

Read Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models? by Renye Yan, Jikang Cheng, Shikun Sun, Yi Sun, You Wu, Wei Peng, Zongwei Wang, Ling Liang, Junliang Xing, Yimao Cai, [0.2cm] Peking University, Tsinghua University, Nanjing University, Stanford University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15855.txt.

Introduction

Renye Yan and nine colleagues argue that running reinforcement learning over the full denoising path of diffusion models wastes effort and weakens preference fit. Early shapes are unstable and far from the final reward, so rewards lag and actions mismatch. Late gains stall, and further training overfits fine details, boosting reward hacking. Their AdaScope plug-in finds the time to intervene by tracking shape change and meaning match, then halts once denoising settles and reward gains stall. Against top rivals, AdaScope lifts performance by 66 percent while cutting cost by 59 percent.

Catalog record

CREATORRenye Yan, Jikang Cheng, Shikun Sun, Yi Sun, You Wu, Wei Peng, Zongwei Wang, Ling Liang, Junliang Xing, Yimao Cai, [0.2cm] Peking University, Tsinghua University, Nanjing University, Stanford University
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51804
LICENSEhttp://creativecommons.org/publicdomain/zero/1.0/
RIGHTSCC0
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15855

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Less, Achieve More: Do We Need Every-Step Optimization for RL Fine-tuning of Diffusion Models? Renye Yan, Jikang Cheng, Shikun Sun, Yi Sun, You Wu, Wei Peng, Zongwei Wang, Ling Liang, Junliang Xing, Yimao Cai, [0.2cm] Peking University, Tsinghua University, Nanjing University, Stanford University Abstract Despite strong image-generation performance, diffusion models’ reconstruction objectives limit alignment with human preferences. RL enables such alignment through explicit rewards. However, most studies apply RL to the full denoising trajectory, making it computationally costly and weakening preference alignment, i.e., doing more but achieving less. We observe that the impact of RL fine-tuning varies significantly across denoising stages. In the early stage, image structures are unstable and distant from the final reward signal. Applying RL at this stage leads to delayed rewards and action–reward mismatching, resulting in high variance and inefficient updates. Conversely, in the later stage, reward gains saturate, and continued training tends to overfit local details, intensifying reward hacking. To tackle these challenges, we propose AdaScope, an RL-enhanced plug-in that improves generation quality while reducing computational cost. Specifically, AdaScope adaptively identifies the optimal intervention timing for RL by perceiving the structural evolution and semantic consistency during denoising, and dynamically terminates training once the denoising converges and reward gains saturate. As a result, it achieves a rare ‘dual benefit’: a reduction in computational costs alongside a significant performance improvement. We offer theoretical grounds for the design of AdaScope. Compared with state-of-the-art methods, AdaScope improves performance by 66% while cutting computational cost by 59%. Figure 1 : We plot the CLIP variation ( \Delta CLIP), Reward Objective, and Uncertainty Score (Based on Lemma 1 ) with aligned denoising steps. Only the red region is optimized, where we leverage \Delta CLIP and Reward to select the adaptive scope of denoising steps for training. It can be observed that the structure is chaotic in the first stage, while the reward converges in the last stage. The selected scope has a stable structure and improving reward, which exactly corresponds to the moderate uncertainty stage. 1 Introduction Diffusion models 49 ; 16 ; 34 ; 38 ; 40 ; 43 ; 5 ; 4 ; 24 ; 55 ; 71 have emerged as the core framework for general-purpose image generation, owing to their powerful distribution modeling capability and exceptional generation quality. By progressively denoising to approximate the true distribution, they can faithfully reconstruct complex multimodal structures and have achieved groundbreaking progress in fields such as vision, speech, and 3D generation 15 ; 17 ; 60 ; 70 . However, the optimization objective of such models primarily focuses on data reconstruction, lacking task-specific or goal-oriented constraints, such as aesthetic preferences 45 or user-specific intents 65 . As a result, although these models possess strong generative potential, they struggle to achieve fine-grained alignment with specific task. Recent studies have attempted to incorporate reinforcement learning (RL) 46 ; 10 ; 69 ; 64 into the diffusion model fine-tuning, enabling reward signals to guide the generation process toward goal-oriented optimization directly 1 ; 8 ; 65 . RL methods 1 ; 35 ; 8 ; 65 ; 53 ; 63 extend the optimization objective from data reconstruction to preference consistency, thereby achieving task-level alignment. However, the introduction of RL inevitably brings the issue of sparse feedback. Since rewards are only available after the final image is generated, the early denoising stage lacks effective guidance signals, making policy optimization difficult. To address this, mainstream approaches typically adopt a straightforward compromise: backpropagating the final reward to all denoising steps to ensure RL functions properly 7 ; 1 ; 9 ; 3 ; 51 . In addition, the large computational overhead introduced by RL fine-tuning is also highly significant 23 ; 58 . Although RL fine-tuning methods have achieved remarkable results, they still face three critical unresolved challenges. (1) Temporal Misalignment Caused by Reward Backfilling: While backpropagating the final reward to all denoising steps mitigates sparse feedback, it disrupts temporal causality. Early steps receive guidance signals weakly correlated with their actual contributions. Moreover, due to the rapid structural changes in the image during the early stage, RL training tends to have high variance and random gradient directions 7 ; 1 ; 9 ; 3 ; 51 . (2) Aggravated Reward Hacking Due to Neglected Stage Sensitivity: Existing methods typically align the training termination point with the final denoising step, implicitly assuming equal optimization value across all stages. This assumption overlooks the fact that structural and feature representations become highly stable in the late stage, where denoising has nearly converged. The marginal benefit of RL training at this stage is minimal. Forcing model updates in such low-value intervals increases the risk of amplifying reward hacking 48 ; 11 ; 68 ; 31 . (3) Negative Impact and Computational Waste: The issues described in (1) and (2) not only hurt final generation quality but also lead to significant computational resource waste 50 ; 23 ; 58 ; 1 ; 7 ; 65 ; 9 . To address the issues above systematically, we propose AdaScope, an RL plugin that adaptively screens training stages. Unlike the current training paradigm of optimizing full trajectories at equal frequency, AdaScope dynamically identifies the optimal intervention and termination points for RL. Specifically, AdaScope uses semantic maturity as a criterion to adaptively determine the starting point for RL, skipping the chaotic stage in which structural formation is incomplete and noise predominates. This helps alleviate reward-attribution mismatch. Subsequently, by jointly modeling denoising completeness and reward-gain trends, AdaScope automatically stops updates when marginal returns saturate, thereby effectively suppressing low-return training and reward hacking. Ultimately, the optimization process focuses on the key interval where semantic structures are stable and training returns are high. Since AdaScope dynamically filters out denoising stages that contain a large number of inefficient or counterproductive samples, it not only optimizes computational resources but also delivers significant improvements in generation quality. We validate the design rationale of AdaScope by visualizing the correlation between adjacent intermediate denoising steps and conducting corresponding theoretical analyses. To verify the integrability and effectiveness of AdaScope, we integrated it with 4 different approaches. The results demonstrate that our method significantly outperforms state-of-the-art methods across 6 metrics while reducing computational costs to 59%. In summary, this work presents the following three key innovations: • We demonstrate the rationality of our method’s design through theoretical derivations of the correlations among intermediate images during the generation process. Furthermore, we provide a visualization analysis of the correlation metric, whose observed results are consistent with the derived conclusions. • In terms of quality, AdaScope automatically filters out early-stage samples with severe reward attribution mismatch, thereby enhancing the stability of policy gradient estimation. Simultaneously, it eliminates late-stage samples that tend to overfit, thereby strengthening generalization and distribution fidelity. • From a computational perspective, AdaScope reduces training steps by adaptively identifying effective RL intervals, thus avoiding ineffective updates in early high-noise and late-saturation stages. This design yields higher returns under the same computation cost. 2 Related Works Sparse Rewards: In RL fine-tuning for diffusion models, reward signals are typically computed precisely only on the final generated image, resulting in a sparse rewards 19 ; 30 ; 13 ; 54 ; 39 ; 12 profile during fine-tuning. The diffusion model

Read the complete text