ROCM: RLHF on consistency models
Shivanshu Shekhar, Tong Zhang · 2025
Read ROCM: RLHF on consistency models by Shivanshu Shekhar, Tong Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06171.txt.
Introduction
Diffusion models make images by slowly cleaning noise, which makes them slow, and tuning them with human feedback is slower still. Consistency models shortcut the process, generating in one or a few steps. Shekhar and Zhang bring human-feedback training to consistency models with a direct reward method they call ROCM. Instead of the usual policy-gradient route, their method follows first-order gradients of the reward.
A spread-based penalty keeps the model from drifting too far or gaming the reward. The team tests several forms of that penalty, drawn from a family of divergence measures. Across automatic scores and human judgments, ROCM matched or beat policy-gradient methods. It also proved steadier and less fussy about settings. The authors present it as an efficient way to align fast generators with human taste.
Catalog record
| CREATOR | Shivanshu Shekhar, Tong Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-06171
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ROCM: RLHF on consistency models Shivanshu Shekhar, Tong Zhang Abstract Diffusion models have revolutionized generative modeling in continuous domains like image, audio, and video synthesis. However, their iterative sampling process leads to slow generation and inefficient training, challenges that are further exacerbated when incorporating Reinforcement Learning from Human Feedback (RLHF) due to sparse rewards and long time horizons. Consistency models address these issues by enabling single-step or efficient multi-step generation, significantly reducing computational costs. In this work, we propose a direct reward optimization framework for applying RLHF to consistency models, incorporating distributional regularization to enhance training stability and prevent reward hacking. We investigate various f f -divergences as regularization strategies, striking a balance between reward maximization and model consistency. Unlike policy gradient methods, our approach leverages first-order gradients, making it more efficient and less sensitive to hyperparameter tuning. Empirical results show that our method achieves competitive or superior performance compared to policy gradient based RLHF methods, across various automatic metrics and human evaluation. Additionally, our analysis demonstrates the impact of different regularization techniques in improving model generalization and preventing overfitting. 1 Introduction Figure 1 : Examples of images generated by the model aligned using the KL divergence regularization constraint and HPS reward model Diffusion models have brought about significant advancements in the modeling of continuous domains, including chemical molecule design [ 35 ] , audio generation [ 12 ] , text-to-image synthesis [ 21 ] , and video generation [ 16 ] . These models have demonstrated remarkable success across various applications, showcasing their versatility and potential. However, a notable challenge with diffusion models is their slow generation process. The iterative nature of the diffusion process means that each sample generation involves multiple steps, often making it difficult to train these models in an end-to-end manner. Researchers usually have to resort to approximations in the learning pipeline [ 1 , 29 ] or endure extensive training times to fine-tune diffusion models effectively [ 3 ] . This challenge becomes even more pronounced when applying reinforcement learning (RL) pipelines to fine-tune diffusion models. In these scenarios, rewards are provided only at the final step of the generation process and as the time horizon increases the resulting sparse reward signals can significantly hinder the training performance. To tackle this problem, we turn our attention to the performance of Reinforcement Learning from Human Feedback (RLHF) when applied to consistency models [ 26 ] . Consistency models, in contrast to diffusion models, offer the advantage of efficient generation with a small number of steps. In practice, they can produce competitive results within 4-8 steps, compared to the 20-50 steps typically required by diffusion models, thus addressing the issue of slow generation to a large extent. A key observation in our study is that simply maximizing the reward in the RLHF pipeline often leads to overfitting and reward hacking, as the trained model diverges significantly from the original model. Reward hacking arises when the generation policy strays too far from the reference model, producing samples that are substantially different from those used to train the reward model. In such an out-of-distribution regime, a high reward does not necessarily indicate high-quality outputs. Over-optimizing for the reward can therefore lead to poor-quality images that receive artificially inflated scores. To counter this, it is common to alter the objective to not only maximize the reward but also to minimize the divergence between the current model and the reference model distributions. The reference model is generally set to the base model at the beginning of training. By experimenting with different f -divergence measures, we find that this form of regularization helps stabilize the training process, preventing the model from degenerating into reward hacking and ensuring more robust performance across various metrics. While prior research has applied methods such as Proximal Policy Optimization (PPO) and its variations to both diffusion and consistency models [ 19 , 29 , 1 , 4 , 33 ] , our work emphasizes that such complex training approaches may not always be necessary. We show that using the reparameterization trick [ 10 ] , we can directly optimize the regularized RLHF objective by backpropagating through the entire generation trajectory. Our experiments consistently demonstrate that direct optimization of the RLHF objective can outperform the use of PPO both in training stability and efficiency. Furthermore, a user study corroborates the effectiveness of our approach, underscoring its potential as a simpler yet robust alternative for training consistency models. Our contributions in this work can be summarized as follows: • We formulate and analyze the role of distributional regularization in RLHF for fine-tuning consistency models, demonstrating its impact on training stability, efficiency and reward alignment. • We reformulate the RLHF optimization problem as a direct optimization objective by leveraging the reparameterization trick, allowing efficient backpropagation through the generation trajectory. This reformulation transforms a zero-order optimization problem into a first-order one, significantly enhancing optimization efficiency. Empirically, our results demonstrate that this approach achieves performance on par with or superior to policy gradient based methods while requiring substantially less hyperparameter tuning and enabling faster training. • We conduct a comprehensive empirical analysis of various f -divergence measures for regularization, highlighting their influence on training stability and model performance. 2 Related Works Diffusion Models: Diffusion models have emerged as a powerful class of generative models for tasks involving modeling of continuous data distributions. Inspired by non-equilibrium thermodynamics, these models learn to reverse a stochastic process that gradually adds noise to data, effectively learning the data distribution by reversing this process during generation [ 23 , 7 ] . The iterative nature of diffusion models, where samples are generated through a sequence of denoising steps, allows them to produce high-quality outputs. However, this multi-step generation process is computationally intensive, leading to long inference times. To address this, recent work has focused on developing more efficient variants, such as DDIM [ 24 ] , which accelerates sampling by reducing the number of steps while maintaining output quality. Despite these advancements, the slow sampling speed of diffusion models remains a significant limitation, particularly when integrated with reinforcement learning frameworks for fine-tuning, as storing gradients for every timestep remains memory-intensive, even when only fine-tuning the LoRA layers [ 8 ] . Consistency Models: Consistency models present an alternative approach to generative modeling by enabling single-step or few-step generation [ 26 ] . These models are trained to maintain consistency in their outputs across multiple forward passes, facilitating much faster sampling compared to traditional diffusion models. The core idea is to train a network capable of directly mapping noise from any point in time to the target data distribution in a single step. For multi-step generation, noise is added at each step to the predicted target distribution sample, followed by the re-application of the denoising network, which allows these models to achieve competitive results in a limited number of steps. This approach drastically reduces the computational overhead during inference, making it particularly advantageous for scenarios requiring rapid generation. The ability of consistency models to produce high-quality samples in just a few steps also makes them well-suited for integration with reinforcement learning frameworks, where efficient feedback is crucial. For instance, RLCM [ 19 ] employs PPO to fine-tune