Adversarial Training of Reward Models
Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao · 2025
Read Adversarial Training of Reward Models by Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06141.txt.
Introduction
Alexander Bukharin and eight co-authors introduced Adv-RM in twenty twenty-five, an adversarial training method for reward models. Reward models score language model outputs for alignment training, but they often give high marks to low-quality, off-distribution text. Policies then hack the reward, finding shortcuts instead of genuinely improving.
Adv-RM trains a reinforcement learning policy whose job is to craft responses that fool a target reward model — high scores for bad text. It reliably exposed weaknesses even in a very large state-of-the-art reward model. Folding those adversarial examples back into training made reward models more robust and downstream alignment training more stable. The method outperformed conventional training on both synthetic and real data.
Catalog record
| CREATOR | Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-06141
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adversarial Training of Reward Models Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao Abstract Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples — responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings. We will open-source all code and data. 1 Introduction As large language models (LLMs) become increasingly capable, ensuring their alignment with human values remains a fundamental challenge ( Amodei et al., 2016 ; Askell et al., 2021 ) . One of the most promising approaches for scalable alignment of LLMs is reward modeling, in which a reward model is trained to predict human feedback ( Christiano et al., 2017 ; Leike et al., 2018 ) . These reward models (RMs) are then used as a proxy for human feedback in a variety of places throughout the LLM lifecycle, including data selection ( Chen et al., 2023 ) , Reinforcement Learning from Human Feedback (RLHF, Ouyang et al. 2022 ), and inference time moderation ( Inan et al., 2023 ) . This approach has been very successful, resulting in powerful language models such as GPT-4 and Llama 3 ( Achiam et al., 2023 ; Dubey et al., 2024 ) . Despite their widespread use in LLM alignment, RMs are imperfect proxies for human judgment ( Hendrycks et al., 2021 ; Gao et al., 2023 ) . In particular, they exhibit poor out-of-distribution (OOD) generalization, often failing to reliably assess prompt-response pairs that diverge from their training distribution ( Eisenstein et al., 2023 ) . OOD generalization is challenging because RMs learn from a limited set of human-annotated examples, which do not cover the full diversity of possible model behaviors, leading to systematic failures when encountering novel model responses. These failures lead to reward hacking—where models trained on RMs optimize reward scores through unintended shortcuts, reducing their actual alignment with human values. Such reward hacking can result in unexpected behaviors and difficult-to-detect misalignment, an issue that becomes even more pernicious as model capabilities increase and evaluation becomes more challenging. Building robust RMs to mitigate reward hacking presents two unsolved challenges. The first is measuring a RM’s robustness. Unlike image and speech models, where adversarial attacks can serve as robustness metrics ( Carlini & Wagner, 2017 ) , the discrete nature of language makes adversarial attacks on language models challenging and their robustness harder to define ( Wang et al., 2021 ) . Reward models introduce another layer of complexity, as their accuracy is measured on preference pairs, requiring adversarial preference pairs rather than a single adversarial input. So far the only known reliable way to quantify robustness of a RM is to measure how long it takes for reward hacking to occur in RLHF. This in itself can be challenging: we can rely on human evaluation to tell us if the policy training has significantly degraded, but this is costly and small misalignments can easily go undetected. The second problem is how to improve the robustness of a RM. Prior works show that reward hacking often occurs because the RM gives high reward scores to model responses outside the reward model training distribution ( Casper et al., 2023 ; Eisenstein et al., 2023 ) . Giving a high reward to OOD responses is bad, as they deviate significantly from the expert responses in the RLHF dataset and are therefore likely to be low quality ( Ho & Ermon, 2016 ) . To avoid rewarding such responses, contemporary works have tried a variety of approaches, including regularizing training so that the policy model remains close to its initialization ( Jaques et al., 2019 ; Stiennon et al., 2020 ) , penalizing out-of-distribution (OOD) model responses using uncertainty estimates ( Eisenstein et al., 2023 ; Coste et al., 2023 ) , and regularizing reward learning ( Ibarz et al., 2018 ; Liu et al., 2024b ) . However, we find that uncertainty estimates are not reliable indicators of response quality for in-distribution responses, which may limit the effectiveness of such regularization. In this work we propose a new adversarial training framework, Adv-RM, to address both problems. Adv-RM automatically discovers adversarial responses that the targeted RM gives a high reward to while being OOD and low quality. We find these examples by training an adversarial policy using reinforcement learning to generate responses that maximize discrepancy across different RMs, while receiving high rewards from the target RM. This approach reliably finds adversarial examples for state-of-the-art (SOTA) RMs, including Skywork-Reward-Gemma-2-27B ( Liu et al., 2024a ) , Llama-3.1-Nemotron-70B-Reward ( Wang et al., 2024 ) , and Nemotron-4-340B-Reward ( Adler et al., 2024 ) , with over 80% attack success rates. Interestingly, we find that these popular RMs give high reward scores to a variety of low-quality responses, such as responses that have no punctuation or responses with random text in the middle of a response. Once we find adversarial samples, Adv-RM incorporates them into the RM training procedure. Since the success rate of our attack method is high, we can include adversarial samples in the training data set without additional human annotation cost by simply using an in-distribution response as the chosen response and the adversarial response as the rejected response. We conduct extensive evaluation of these adversarially trained RMs through adversarial attacks, downstream RLHF training in the synthetic setting of Gao et al. (2023) , and RLHF training on real datasets ( Wang et al., 2024 ) . We found that Adv-RM reward models not only are more robust against attacks but also mitigate reward hacking. With these models, RLHF can proceed for three times as many steps as runs with conventional RMs without exhibiting reward hacking, resulting in more performant aligned models. We summarize our contributions as follows: \bullet We develop an RL-based approach that successfully generates adversarial samples for SOTA RMs. To the best of our knowledge, this is the first work to do so without human domain knowledge, providing a new means of evaluating RM robustness. \bullet We use adversarial samples to augment RM training. Through extensive experiments, we find that such adversarial training results in reward models that are more robust to hacking and have better downstream performance in RLHF. 2 Related Work Mitigating Reward Hacking. There have been several works that attempt to mitigate reward hacking. Some of the earliest works used reward ensembles, and found that different ensembling approaches can slow (but not