CoCA: Step-Level Reward for Free in RL-Finetuned T2I Diffusion Models

Xinyao Liao, Wei Wei, Xiaoye Qu, Qiyuan He, Angela Yao, Yu Cheng · 2025

Read CoCA: Step-Level Reward for Free in RL-Finetuned T2I Diffusion Models by Xinyao Liao, Wei Wei, Xiaoye Qu, Qiyuan He, Angela Yao, Yu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19196.txt.

Introduction

Xinyao Liao and co-workers found free step-by-step feedback for training image generators in a 2025 paper. Reinforcement learning fine-tunes text-to-image diffusion models toward human preference, but the reward arrives only once, at the end of each run. That leaves the method guessing which denoising steps deserved credit. Their setup, CoCA, instead tracks how the cosine similarity between each intermediate image and the final one changes. It uses those changes to spread the reward across steps in proportion to each step's real contribution.

No extra preference network is needed, and the optimal policy of the first formulation is preserved. Across four human preference reward functions, the method improved sample speed by well over twofold in the best cases, and it spread to new cases better. Dense credit assignment turns a sparse guessing game into guided learning.

Catalog record

CREATORXinyao Liao, Wei Wei, Xiaoye Qu, Qiyuan He, Angela Yao, Yu Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-19196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoCA: Step-Level Reward for Free in RL-Finetuned T2I Diffusion Models Xinyao Liao, Wei Wei, Xiaoye Qu, Qiyuan He, Angela Yao, Yu Cheng Abstract State-of-the-art text-to-image diffusion models often leverage reinforcement learning for fine-tuning, but they suffer from severe reward sparsity due to the lack of reliable intermediate evaluation. Since existing methods typically receive only a single terminal reward, they fail to identify which specific denoising steps meaningfully contribute to final image quality. To address this, we propose CoCA, a simple and general contribution-based credit assignment framework that redistributes the terminal reward across the denoising trajectory. CoCA estimates each step’s contribution by measuring the latent progression toward the clean image, enabling dense and informative rewards without requiring auxiliary networks. Our approach integrates seamlessly with various policy-gradient algorithms, such as REINFORCE and GRPO, through reward shaping. Empirical results demonstrate that CoCA improves optimization efficiency by 25 to 100%, while preserving the original objective under policy optimization. 1 Introduction Figure 1 : Reward shaping challenge in diffusion denoising. (1) Top: intermediate predicted images from t{=}50 to t{=}0 show the global structure restored in the early stage and fine-grained details refined in later steps. (2) Bottom-left: the aesthetic score increases as denoising proceeds but plateaus in later steps. (3) Bottom-right: the step-wise reward gain rapidly diminishes over time, indicating that later denoising steps contribute far less to final image quality. Figure 2 : Overview of trajectory-level reward and step-level credit assignment. (I) Trajectory-level sparse rewards treat all timesteps uniformly, offering no guidance on where improvements occur. (II) CoCA converts this scalar reward into dense, step-aware credit by weighting each timestep according to its measured contribution. Visualizing the reward gains ( \Delta HPSv2 ( Wu et al., 2023 ) ) across timesteps shows that CoCA accelerates the formation of coherent global structure earlier in the denoising process. Diffusion models ( Sohl-Dickstein et al., 2015 ; Ho et al., 2020 ; Song et al., 2020 ; Tian et al., 2025 ) have emerged as the dominant paradigm in image generation, offering high image fidelity and easy scalability. Recent advances in text-to-image (T2I) diffusion models ( Rombach et al., 2022 ; Betker et al., 2023 ; Chang et al., 2024 ) have greatly expanded the realism and expressiveness of visual content generation. However, despite strong pretrained capabilities, their outputs may not fully align with human preference. In particular, characteristics like aesthetic appeal and subtle perceptual qualities tend to fall outside strict semantic correctness. This gap has motivated a growing interest in modeling human preference ( Wu et al., 2023 ; Xu et al., 2023 ) and post-training diffusion models for alignment ( Rafailov et al., 2023 ; Black et al., 2024 ; Xue et al., 2025 ) . To address this alignment gap, recent work explores post-training strategies that directly optimize diffusion models using human feedback. Methods based on differentiable reward models ( Xu et al., 2023 ; Zhang et al., 2024a ) enable end-to-end optimization, though they require rewards to be differentiable and incur substantial memory overhead. DPO-style preference optimization ( Rafailov et al., 2023 ; Wallace et al., 2024 ; Zhong et al., 2024 ) offers a simpler training pipeline but typically yields only modest gains in visual fidelity. In contrast, reinforcement learning (RL) ( Black et al., 2024 ; Fan et al., 2023 ; Xue et al., 2025 ) has emerged as a promising paradigm, as it does not require differentiable rewards and can be optimized with continuous signals. Recent work views diffusion as a policy model ( Black et al., 2024 ; Fan et al., 2023 ) , formulating iterative denoising as a Markov decision process. Under this formulation, policy-gradient algorithms ( Sutton et al., 1999 ; Schulman et al., 2017 ; team, 2025 ) can be applied to optimize the model with reward functions derived from human feedback. These approaches demonstrate improved alignment with prompts and better quality perceived by humans. However, the denoising trajectory of generating images typically shows diminishing marginal improvement in reconstruction quality at later timesteps ( Rombach et al., 2022 ; Li et al., 2024b ; Huang et al., 2024 ; Qian et al., 2024 ; Huang et al., 2025 ) . As shown in Figure 1 , early timesteps play a decisive role in determining the global structure of the image, while later steps contribute mainly to fine-grained details. However, existing policy gradient methods typically apply a sparse reward signal only at the end point and update the policy equally across all timesteps, as shown in Figure 2 . This leads to a mismatch between the similar magnitude of policy updates and the unequal importance of different timesteps. Consequently, existing RL-based methods suffer from suboptimal sample efficiency and limited precision in attributing rewards to specific generative actions ( Harutyunyan et al., 2019 ; Sutton, 1984 ) . Recent efforts to address this mismatch apply fixed temporal discounting to prioritize early denoising steps ( Yang et al., 2024b ) , or train auxiliary networks to predict step-wise critics or preferences ( Zhang et al., 2024b ; Liang et al., 2024 ; Zhang et al., 2025 ) . Crucially, existing methods either impose a priori assumptions about step importance or require costly reward model expansion to mitigate reward sparsity. This work poses a fundamental question: Can we quantify the contribution of each denoising step to the final image quality? To address this, we propose a novel framework that enables Co ntribution-based C redit A ssignment ( CoCA ) for diffusion-based text-to-image generation as shown in Figure 2 (II). Specifically, for each denoising step, we track similarity progression between the intermediate latent and the realized terminal latent in embedding space, yielding an outcome-aligned latent-progress proxy. We use this proxy to redistribute the terminal reward into denser learning signals, keeping the optimization target aligned with the same terminal objective rather than introducing any additional per-step objective. Then, we convert the sparse trajectory-level reward into informative step-level rewards by redistributing the terminal reward according to the estimated step contributions, without training additional networks. To further stabilize training, we normalize rewards once before redistribution to maintain per-prompt ranking, and once after redistribution to keep reward scales consistent across timesteps and reduce variance during optimization. Theoretically, we show that this reward redistribution does not bias the expected policy gradient, and thus preserves the policy-gradient stationary points of the original objective. Experiments across four reward functions based on DDPO ( Black et al., 2024 ) demonstrate that our framework achieves 25%-100% improvement of sample efficiency than trajectory-level reward baselines ( Black et al., 2024 ; Xue et al., 2025 ) and step-level reward baselines ( Zhang et al., 2024b ) . We also apply CoCA to DanceGRPO ( Xue et al., 2025 ) , the state-of-the-art GRPO ( team, 2025 ) framework in T2I diffusion. The quantitative study shows that our method exhibits better prompt alignment and improves the generalization capability on both unseen rewards and unseen prompts, all without sacrificing computational simplicity, demonstrating rapidly changing the global layout. To summarize, our contributions are as follows: (1) We introduce CoCA, a novel credit assignment framework to deal with the mismatch between equal policy updates and the varying impact of steps caused by reward sparsity, without

Read the complete text