Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling

Pankayaraj Pathmanathan, Furong Huang · 2025

Read Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling by Pankayaraj Pathmanathan, Furong Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06419.txt.

Introduction

Pankayaraj Pathmanathan and Furong Huang teach reward models to hunt their own blind spots. Their 2025 paper addresses the models that score AI answers by human preference and steer larger models, yet crack under odd inputs or sly phrasing.

Their method needs no advance knowledge of where the cracks are. The reward model itself guides a generator toward answers it scores wrongly, harvesting its own failure cases. Those cases join the training data, patching the model's behavior. The framework, called REFORM, held up on two standard preference data sets: direct scores stayed strong, models trained against the patched judge aligned better, and false correlations faded. The authors offer self-found failures as a route to sturdier preference modeling.

Catalog record

CREATORPankayaraj Pathmanathan, Furong Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06419

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling Pankayaraj Pathmanathan, Furong Huang Abstract Reward modeling (RM), which captures human preferences to align large language models (LLMs), is increasingly employed in tasks such as model fine-tuning, response filtering, and ranking. However, due to the inherent complexity of human preferences and the limited coverage of available datasets, reward models often fail under distributional shifts or adversarial perturbations. Existing approaches for identifying such failure modes typically rely on prior knowledge about preference distributions or failure attributes, limiting their practicality in real-world settings where such information is unavailable. In this work, we propose a tractable, preference-distribution-agnostic method for discovering reward model failure modes via reward-guided controlled decoding. Building on this, we introduce REFORM, a self-improving reward modeling framework that enhances robustness by using the reward model itself to guide the generation of falsely scored responses. These adversarial examples are then used to augment the training data and patch the reward model’s misaligned behavior. We evaluate REFORM on two widely-used preference datasets—Anthropic Helpful-Harmless (HH) and PKU Beavertails—and demonstrate that it significantly improves robustness without sacrificing reward quality. Notably, REFORM preserves performance both in direct evaluation and in downstream policy training, and further improves alignment quality by removing spurious correlations. Code: https://github.com/pankayaraj/REFORM Figure 1 : Failure mode detection as controlled decoding: We generate mis-specified responses—e.g., low-scoring preferred responses—by guiding a preference-aligned policy to produce aligned outputs that are adversarially optimized to receive low rewards from the reward model. 1 Introduction Reward modeling (RM) from human preferences plays a central role in the alignment of large language models (LLMs). While recent approaches such as Direct Preference Optimization (DPO) ( Rafailov et al. 2024 ) allow models to be aligned directly from preferences without explicitly learning a reward function, explicit reward modeling remains critical for improved out-of-distribution (OOD) generalization ( Xu et al. 2024 ) . Moreover, reward models are widely reused in downstream applications, including response ranking, test-time alignment ( Mudgal et al. 2024 ) , and reward-guided generation ( Xu et al. 2025 ) . Despite their utility, reward models often fail under perturbations ( Jiang et al. 2025 ) . These failures stem from the inherent complexity of human preferences and the limited perturbation coverage of preference datasets. As a result, reward models are prone to mis-specification—assigning incorrect rewards to preferred or non-preferred responses—which can lead to over-optimization or reward hacking ( Ramé et al. 2024 , Hendrycks et al. 2022 , Park et al. 2024 , Zeng et al. 2024 ) . Identifying such failure modes not only helps diagnose reward model behavior but also offers a path to improving reward robustness. In this work, we define failure modes as variants of preferred or non-preferred responses that are incorrectly scored—e.g., a preferred response receiving a lower reward than its negative counterpart. Previous work by Jiang et al. 2025 casts reward model analysis as a counterfactual generation task, prompting LLMs such as GPT-4 ( OpenAI et al. 2024 ) to generate contrastive examples conditioned on interpretable attributes. While an improvement over earlier approaches such as Wu et al. 2021 , these methods require prior knowledge of reward-relevant attributes—an impractical assumption in many settings. Reward behavior often depends on factors beyond the preference distribution, including the model architecture, training method etc. Thus, these methods fail to generalize in model- or distribution-agnostic contexts. Research Objective. Motivated by these limitations in both reward modeling and failure analysis, we ask: Can reward model failure modes be tractably identified in a preference-distribution-agnostic way, and can the reward model improve itself using these failure modes to become more robust? We present a solution to this question through the following three research components: RQ1: Can failure modes be discovered tractably and without preference knowledge? We introduce a controlled decoding framework that leverages the reward model itself to uncover mis-specified examples—responses that are incorrectly scored relative to their known preference class. Specifically, a failure mode must satisfy two conditions: (W1) the response must have a clearly identifiable class with respect to the preference distribution (i.e., unambiguously preferred or non-preferred), and (W2) the reward model assigns a score that contradicts this class (e.g., assigning low reward to a preferred response or high reward to a non-preferred one). To satisfy W1 , we begin with a policy that is either aligned or misaligned with the target preference distribution—obtained via supervised training or test-time alignment using the reward model. We use this policy to generate examples that reliably reflect a known preference class. To satisfy W2 , we apply reward-guided controlled decoding to steer generation toward responses that the reward model scores incorrectly. This procedure yields class-consistent yet reward-inconsistent examples—i.e., failure modes. Empirically, we show that this approach discovers such failure modes more effectively than model-agnostic counterfactual baselines. (a) Data augmentation (b) Reward correction Figure 2 : Robust reward modeling via self-improvement: Failure-mode examples are generated via controlled decoding as in Figure 1 are used to augment the training dataset as seen in Figure 2a , leading to a more robust reward model as demonstrated in Figure 2b . RQ2: Does training the reward model with awareness of these failure modes improve robustness? A key advantage of the discovered failure modes is that they represent perturbations outside the original training distribution, offering a valuable opportunity to enhance reward model robustness. We show that these failure-mode examples can be used to fine-tune the reward model, making it more resilient to such perturbations. Specifically, we construct an augmented preference dataset by identifying failure variants of only the most influential 5% of prompt-response pairs in the original training set and correcting them. We define a failure variant as follows: for a given preference pair (positive vs. negative), if a modified version of the positive response is scored lower than the corresponding negative, or if a modified negative is scored higher than its positive counterpart, the variant is marked as a failure case. By fine-tuning the reward model on these targeted corrections, we substantially improve its robustness without requiring large-scale data augmentation or manual relabeling. RQ3: Does this robustness come at the cost of reward model performance? We demonstrate that the reward model trained with failure-mode awareness (REFORM) maintains its performance on standard, unperturbed test data, while also exhibiting improved robustness. Crucially, this robustness does not come at the expense of downstream alignment quality. Across three popular alignment methods—Best-of-N sampling (BoN), Proximal Policy Optimization (PPO), and Direct Preference Optimization (DPO)—REFORM preserves generation quality in terms of diversity, readability, and user utility. These results indicate that our approach achieves robustness without compromising alignment performance, and may even enhance it by mitigating spurious correlations introduced by mis-specified rewards. 2 Related Works 2.1 Reward hacking in LLMs Reward models tend to show various limitations in attributing scores to preferences due to several reasons, such as model size, distribution shift in data, and the lack of representation of preferences in data ( Eisenstein et al. 2024 , Gao et al. 2022 , Guo et al. 2024 , Xiong et al. 2024 ) . Some lines of work have explicitly identified artifact-based biases, such as length in reward models and explicitly designed regularization methods ( Chen et al. 2024

Read the complete text