Does RLHF Scale? Exploring the Impacts from Data, Model, and Method
Zhenyu Hou Pengfan Du Yilin Niu Zhengxiao Du Aohan Zeng Xiao Liu, Minlie Huang Hongning Wang Jie Tang Yuxiao Dong, [0.5ex] Tsinghua University Zhipu AI[0.5ex] · 2024
Read Does RLHF Scale? Exploring the Impacts from Data, Model, and Method by Zhenyu Hou Pengfan Du Yilin Niu Zhengxiao Du Aohan Zeng Xiao Liu, Minlie Huang Hongning Wang Jie Tang Yuxiao Dong, [0.5ex] Tsinghua University Zhipu AI[0.5ex] (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06000.txt.
Introduction
Teaching language models from human feedback is standard practice, but nobody knew how well it scales. Zhenyu Hou and co-authors mapped in 2024 what happens as data, model size, and method choices grow.
More diverse feedback data improved the reward models that grade answers, and helped step-by-step supervision most. For the final training, sampling more replies per prompt helped briefly, then flattened. Bigger reward models gave only modest gains, and bigger policy models gained less from a fixed grader. Overall, feedback training scaled worse than raw pretraining. The authors close with strategies for squeezing the most from a limited compute budget.
Catalog record
| CREATOR | Zhenyu Hou Pengfan Du Yilin Niu Zhengxiao Du Aohan Zeng Xiao Liu, Minlie Huang Hongning Wang Jie Tang Yuxiao Dong, [0.5ex] Tsinghua University Zhipu AI[0.5ex] |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06000
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Does RLHF Scale? Exploring the Impacts from Data, Model, and Method Zhenyu Hou Pengfan Du Yilin Niu Zhengxiao Du Aohan Zeng Xiao Liu, Minlie Huang Hongning Wang Jie Tang Yuxiao Dong, [0.5ex] Tsinghua University Zhipu AI[0.5ex] Abstract This study explores the scaling properties of Reinforcement Learning from Human Feedback (RLHF) in Large Language Models (LLMs). Although RLHF is considered an important step in post-training of LLMs, its scaling potential is still largely unknown. We systematically analyze key components in the RLHF framework—model size, data composition, and inference budget—and their impacts on performance. Our findings show that increasing data diversity and volume improves reward model performance, helping process-supervision models scale better. For policy training, more response samples per prompt boost performance initially but quickly plateau. And larger reward models offer modest gains in policy training. In addition, larger policy models benefit less from RLHF with a fixed reward model. Overall, RLHF scales less efficiently than pretraining, with diminishing returns from additional computational resources. Based on these observations, we propose strategies to optimize RLHF performance within computational limits. 1 Introduction Large Language Models (LLMs) have revolutionized natural language processing by learning extensive language patterns from vast datasets. A key step in these models is reinforcement learning from human feedback (RLHF) ( Ouyang et al. 2022 ) , which helps align the model’s behavior with human intentions and enhances their performance across diverse tasks such as text generation Hu et al. 2024 , coding ( Li et al. 2022 ; Zhu et al. 2024 ) , and mathematical reasoning ( Zhu et al. 2024 ) before deployment. This approach has been successfully applied to leading models like ChatGPT ( Achiam et al. 2023 ) , Llama ( Touvron et al. 2023 ; Dubey et al. 2024 ) , and Claude ( Bai et al. 2022b ) , yielding notable improvements. RLHF improves the model’s behavior by integrating external feedback, such as human preferences and answer supervision, to enhance its generation. The process begins with training a reward model using human-labeled preference data or reasoning data with correctness labels, serving as a proxy for human supervision. Then, reinforcement learning is employed to optimize the policy model through iterative feedback from the reward model. The scaling properties of large language models, which lead to considerable performance improvements with increased computational resources (i.e., more data and larger models), have been viewed as key to the success of the current LLMs. However, the scaling properties of RLHF have been less understood, while extensive studies have been conducted for both the pretraining stage ( Kaplan et al. 2020 ; Hoffmann et al. 2022 ; Du et al. 2024 ) and supervised fine-tuning ( Yuan et al. 2023 ) . Recently, the emergence of OpenAI-o1 ( openai 2024 ) has demonstrated the potential of scaling reinforcement learning for reasoning tasks, but the specific methods used to achieve this result have not been disclosed. Previous works ( Ivison et al. 2024 ; Xu et al. ) , including Llama2 ( Touvron et al. 2023 ) , have attempted to explore the greater potential of RLHF, i.e., PPO ( Schulman et al. 2017 ) , compared to DPO ( Rafailov et al. 2024b ) , and Llama3 ( Dubey et al. 2024 ) only describe its use of DPO for optimization. They paid little attention to the scaling potential and properties of current RLHF and there is still a limited practical understanding of large-scale RLHF training. This raises questions about whether current RLHF techniques can lead to significant performance improvement like OpenAI-o1, given access to more data and computing resources, or whether we have to develop a more scalable reinforcement learning algorithm for LLMs. In this work, we systematically investigate the key components of the current RLHF framework that can be scaled in policy model and reward model training, including model size, data composition, and the effect of inference budget. We explore how these components impact the performance of each model individually and the final performance in on-policy RLHF methods, specially PPO ( Schulman et al. 2017 ) and GRPO ( Shao et al. 2024 ) . Since the performance of the policy model is heavily influenced by the preceding pretraining and supervised fine-tuning (SFT) stages, we aim to investigate two problems: (1) Given a fixed SFT model, how does scaling other factors affect the policy model through RLHF? (2) With a fixed reward model and training strategy, whether a larger policy model can benefit more from RLHF? Starting with an initial SFT model, we train more than 20 models with reward and policy model sizes of 9B, 32B, and 200B parameters across varying dataset sizes. We conduct a comprehensive study of different factors and show details of the training process and evaluation results to help us understand what scales in RLHF training. In particular, we pay more attention to reasoning tasks, which are considered more scalable in previous works ( openai 2024 ; Yuan et al. 2023 ) but also conduct experiments on general chat tasks. (a) (b) (c) Figure 1: Left : Performance trend with different reward models and sampled responses in RLHF training using a fixed SFT model. Middle : Performance gain of different sizes of policy model after RLHF. Right : Overview of reinforcement learning from human feedback (RLHF) Observations. Figure 1 highlights our key findings about the scaling property of RLHF on reasoning tasks, and our important observations are summarized in the following: 1. Sampling more responses per prompt during PPO training generally improves policy model’s final performance, but the benefits plateau quickly (Cf. Figure 1a ). 2. Larger reward models can effectively boost performance, but the improvement still significantly falls behind the gains in Best-of- N evaluation of the reward model (Cf. Figure 3 ). 3. Larger policy models benefit less from RLHF when using a fixed size reward model (Cf. Figure 1b ). 4. Performance improves remarkably in the early stage of policy training, but additional data yields only marginal gains despite increasing training rewards (Cf. Figure 5 ). 5. Increasing training data for reward model improves its performance in Best-of- N evaluation, with increasing prompt diversity being more effective than increasing response diversity (Cf. Figure 6 ). 6. Process supervision from automated labeling yields better performance than outcome supervision on the targeted task, but struggles to generalize to other tasks (Cf. Figure 6 ). Overall, our empirical observations suggest that the current RLHF framework does not scale as effectively as the pretraining stage. Increased computational resources do not consistently yield significant and observable performance improvements. This limitation may stem from inaccuracies in the learned reward model or current policy optimization strategies. Further research is deserved to unlock the full potential of reinforcement learning for post-training of LLMs. Nonetheless, our empirical study still suggests some recipes for maximizing the benefits from increased compute within the current RLHF framework. Starting from the baseline of sampling one response per prompt, the following strategies can be considered in order for optimizing performance as compute expands: • Sampling more responses per prompt during policy training, around 4 or 8 for efficiency, is cost-effective for most tasks. Expanding the reward model size is cost-effective, too. • Collect as many