Explicit Critic Guidance for Aligning Diffusion Models
Zhengyang Liang, Qihang Zhang, Ceyuan Yang · 2026
Read Explicit Critic Guidance for Aligning Diffusion Models by Zhengyang Liang, Qihang Zhang, Ceyuan Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27736.txt.
Introduction
Zhengyang Liang, Qihang Zhang, and Ceyuan Yang align diffusion models with an explicit critic guidance framework. Their state-aligned latent actor-critic lets the diffusion model serve as its own timestep-conditioned value function, predicting values directly on noisy latent states. Trajectory-level PPO training plus stable actor-critic optimization and multi-reward training curb reward hacking, and test-time steering adds generation gains.
Catalog record
| CREATOR | Zhengyang Liang, Qihang Zhang, Ceyuan Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57341 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27736
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Explicit Critic Guidance for Aligning Diffusion Models Zhengyang Liang, Qihang Zhang, Ceyuan Yang Abstract Online reinforcement learning is becoming increasingly important for aligning diffusion models with non-differentiable objectives. However, existing methods still face limitations in assigning fine-grained credit along denoising trajectories and in realizing stable value-based optimization. We propose a state-aligned latent actor-critic framework for diffusion post-training, in which the diffusion model serves as its own timestep-conditioned value function and predicts values directly on noisy latent states. This enables trajectory-level PPO training, supports stable actor-critic optimization with simple conditioning and value pretraining strategies, and naturally allows the learned critic to be reused for inference-time steering. We further extend the framework to multi-reward optimization, where joint training with complementary rewards helps alleviate reward hacking. Across both UNet- and DiT-based backbones, our method consistently outperforms prior group-relative RL and actor-critic baselines on single-reward and multi-reward benchmarks, while test-time steering provides additional gains in generation quality. 1 Introduction Post-training is becoming an increasingly important stage for visual generative models. Although large-scale pretraining [ 11 , 61 , 74 , 37 , 84 ] already yields strong visual quality, pretrained models often remain suboptimal on properties that users directly care about, including prompt faithfulness, compositional correctness [ 17 , 24 ] , text rendering [ 6 ] , aesthetics, and human preference [ 77 , 47 , 79 , 27 , 72 ] . Reinforcement learning [ 60 , 1 , 35 , 63 , 92 , 58 , 23 , 51 , 70 , 2 ] is particularly appealing in this setting because it enables optimization with respect to verifiable and task-specific objectives, including learned preference models and rule-based rewards. Figure 1 : Text-conditioned image generation results from our SD3.5M models. All samples are generated without classifier free guidance. A central challenge in diffusion post-training is credit assignment . Diffusion models [ 21 , 68 , 69 , 8 , 40 , 45 ] generate samples through multi-step denoising trajectories, whereas rewards are typically defined only on the final decoded image. The problem is therefore not only to maximize terminal reward, but to assign it to the intermediate noisy states and denoising decisions that produced it. Group-relative methods [ 43 , 81 , 7 ] partially avoid this issue by comparing samples within a group, but the resulting signal remains coarse and trajectory-level. Explicit actor-critic methods are in principle better suited to this setting, yet existing diffusion critics [ 91 , 48 ] are typically defined in pixel space: they evaluate reconstructed clean-image proxies rather than the actual noisy latent states visited by the policy. This mismatch weakens alignment between critic and policy, makes value estimation brittle for structured or discrete rewards, and incurs substantial decoding overhead. Moreover, explicit actor-critic training is often unstable, as poorly conditioned value estimates can quickly undermine policy optimization. To address these issues, we propose a state-aligned latent actor-critic framework in which the diffusion model itself serves as a timestep-conditioned value network and predicts V_{\phi}(z_{t},t,y) directly on noisy latent states. This requires only lightweight modifications to the pretrained diffusion backbone, allowing the critic to inherit the model’s generative prior while operating on the same state space and diffusion-native features as the policy. The resulting critic is naturally noise-aware, avoids repeated image decoding, and can be reused at inference time, where its latent-space gradients provide a direct steering signal during sampling. To make this formulation practical, we further identify timestep-aware conditioning and a short value pretraining stage as simple but effective techniques for improving training stability and efficiency. As illustrated in Fig. 1 , the framework also enables CFG-free post-training, reducing memory overhead and training cost while avoiding the color saturation artifacts and instability often introduced by classifier-free guidance. A stronger optimizer, however, also makes reward misspecification more visible. In diffusion post-training, reward hacking often appears as shortcut solutions that improve the target metric while degrading unmeasured aspects of image quality. For example, optimizing a compositional reward in isolation may improve the target score while sacrificing background diversity or aesthetics, whereas complementary rewards can counteract such collapse modes. This motivates our multi-reward extension: we equip the shared latent backbone with separate value heads so that multiple rewards can regularize one another while still benefiting from a common timestep-aware representation. In this way, multi-reward optimization provides a practical mechanism for mitigating proxy overoptimization in diffusion RL. In summary, our contributions are as follows: • We propose a state-aligned latent critic framework for diffusion post-training, which converts a pretrained diffusion model into its own value network with lightweight modifications and supports both latent-state value prediction and inference-time steering. • We identify timestep-aware conditioning and value pretraining as simple but effective techniques for stabilizing and accelerating explicit actor-critic diffusion training. • We extend the framework to multi-reward optimization, where joint training with complementary rewards helps mitigate proxy overoptimization and alleviate reward hacking. • Across both single-reward and multi-reward benchmarks, our method consistently outperforms prior group-relative RL and actor-critic baselines, with additional gains from inference-time steering. 2 Related Work Diffusion alignment. Recent work on aligning diffusion models can be broadly grouped into reward-based fine-tuning, preference optimization, and online reinforcement learning. Early approaches align text-to-image models with human or learned feedback by collecting preference data, training reward models, and fine-tuning the generator with reward-based objectives [ 30 , 12 , 53 , 80 ] . For differentiable rewards, alignment can also be performed by backpropagating reward gradients through the denoising process [ 79 , 5 , 54 , 55 ] . Another line of work adapts preference optimization methods to diffusion models, including DPO-style formulations [ 57 , 78 , 44 , 36 , 33 , 88 , 87 , 15 , 82 ] . More recently, online RL methods [ 4 , 14 , 60 , 20 , 49 , 91 , 48 , 90 ] optimize diffusion models directly against downstream rewards. This line has also been extended to Group-Relative style policy gradient method [ 43 , 81 , 34 , 32 ] , which bypass the training of critic. Compared with these methods, our work focuses on value-based post-training with an explicit critic learned on the same noisy latent trajectory as the policy. We provide extended discussion in Appendix D . 3 Method 3.1 Problem setup Figure 2 : Illustration of three credit assignment strategies in diffusion RL. GRPO-style methods assign one relative signal to each trajectory by groupwise reward comparison. Pixel-space critics score reconstructed images decoded from intermediate latents, leading to proxy-state evaluation. Our method instead learns a critic directly on noisy latent states, enabling state-dependent and state-aligned credit assignment as well as inference-time steering from the same learned critic. We briefly define the sequential decision-making view of diffusion post-training used throughout the paper. Let z_{t} denote the noisy latent at timestep t , and let y be the text condition. A latent diffusion or flow-based generator produces an image by iteratively updating z_{t} along a denoising trajectory from noise to data. We view this process as a finite-horizon sequential decision problem. At step k , the state is s_{k}=(z_{t_{k}},t_{k},y), and the action a_{k} is the model prediction used by the sampler, e.g., a noise, score, or velocity prediction depending