Modification-Considering Value Learning for Reward Hacking Mitigation in RL
Evgenii Opryshko, Umangi Jain, Igor Gilitschenski · 2026
Read Modification-Considering Value Learning for Reward Hacking Mitigation in RL by Evgenii Opryshko, Umangi Jain, Igor Gilitschenski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28955.txt.
Introduction
Opryshko, Jain and Gilitschenski present a defence against reward hacking, where a learning agent games its reward signal instead of doing its intended task. Standard practical defences keep the agent's updates close to a safe reference policy. That creates a tension: tighter leashes suppress hacking but also block legitimate improvement beyond the reference.
Their method, MCVL, wraps an existing off-policy learner and inspects each new experience before admitting it into training. For every candidate transition, it forecasts two training paths, one including the data point and one excluding it. A frozen estimator, built from a value function and a learned reward model, scores both paths. A transition enters training only if including it does not lower the score. The authors state when this filter is both safe and permissive. Across gridworlds and modified continuous-control tasks with varied hacking tricks, MCVL curbed hacking while still improving the intended objective.
Catalog record
| CREATOR | Evgenii Opryshko, Umangi Jain, Igor Gilitschenski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28955
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modification-Considering Value Learning for Reward Hacking Mitigation in RL Evgenii Opryshko, Umangi Jain, Igor Gilitschenski Abstract Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacking. Existing practical defenses typically constrain policy updates to stay near a known safe reference, creating a tension between suppressing hacking and permitting legitimate improvement. We propose Modification-Considering Value Learning (MCVL), which operationalizes the theoretical idea of current utility optimization for standard value-based RL. MCVL wraps an off-policy learner and treats each incoming transition as a candidate modification: it forecasts two training paths, one that includes the transition and one that does not, and scores both with a frozen bootstrapped-return estimator derived from a learned reward model and value function. The transition is admitted only if inclusion does not decrease the score. We formalize conditions under which this filtering is both safe and permissive, and instantiate MCVL with DDQN and TD3. Across four safety-relevant gridworlds and three modified MuJoCo continuous-control tasks with diverse hacking mechanisms, MCVL mitigates reward hacking while continuing to improve the intended objective. Project website: ktolnos.github.io/mcvl/ . 1 Introduction Optimizing poorly defined or incomplete rewards can push RL agents toward unintended behaviors, leading to reward hacking ( Skalse et al. 2022 ) . For instance, an agent tasked with stacking blocks may learn to flip blocks if the reward is based on the height of the bottom face ( Popov et al. 2017 ) . As RL systems scale to safety-critical applications (e.g., autonomous driving ( Kiran et al. 2021 ) or medical diagnostics ( Ghesu et al. 2017 ) ), ensuring reliable and safe behavior becomes increasingly important. Reward hacking can become more prevalent as models grow in complexity ( Pan et al. 2022 ) , which also affects large language models where RL is used for post-training ( Denison et al. 2024 ; OpenAI 2024 ; MacDiarmid et al. 2025 ) . A common mitigation constrains policy updates around a trusted reference ( Laidlaw et al. 2024 ) , often at a cost to optimality. A complementary safeguard is to optimize what the agent currently values while being conservative about changing those values, an idea discussed as current utility optimization ( Orseau & Ring 2011 ; Hibbard 2012 ; Everitt et al. 2016 ; Everitt et al. 2021 ) . None of these works, however, provides a practical evaluation of this concept. We address this gap by investigating whether individual transitions can be predictive of reward hacking in the context of value-based RL. Our method, Modification-Considering Value Learning (MCVL) , wraps a standard off-policy learner and treats each update as a candidate modification. For a newly observed transition, the agent forecasts two scenarios: one in which it learns from the transition and one that ignores it. Then MCVL scores both resulting policies using its current learned return estimator, an n -step bootstrapped return combining a learned reward model with a value-function bootstrap, and accepts the transition only if inclusion does not decrease this score. MCVL blocks updates that, according to the agent’s current return estimator, would shift behavior toward undesirable strategies. To study it empirically, we instantiate MCVL with DDQN and TD3. Our method only assumes a seed dataset with non-hacking transitions so that the evaluator can identify the intended behavior; for gridworlds we collect this in a Safe variant with the hacking affordance removed, while for continuous control a random-policy dataset suffices ( Section 3 ). Under these conditions, MCVL mitigates reward hacking in multiple safety-relevant gridworlds ( Leike et al. 2017 ; Everitt et al. 2021 ) and modified Gymnasium continuous-control environments ( Towers et al. 2024 ) (Reacher, Ant, HalfCheetah) which we introduce to enable reward-hacking research in continuous control. All code will be made publicly available. 2 Notation and Preliminaries We denote a Markov decision process (MDP) by (S,A,P,R,\rho,\gamma) with state space S , action space A , transition model P(s^{\prime}|s,a)\in[0,1] , reward function R:S\times A\rightarrow\mathbb{R} , initial state distribution \rho , and discount factor \gamma\in(0,1) . For any reward function r , we write J_{r}(\pi)=\mathbb{E}_{\rho,\pi}[\sum_{t\geq 0}\gamma^{t}r(s_{t},a_{t})] for the expected return of the policy \pi under r . In standard RL, the agent’s training objective is to learn a policy \pi maximizing J_{R}(\pi) . The state-action value Q^{\pi}(s,a) is the expected return starting from (s,a) and following \pi thereafter ( Sutton & Barto 2018 ) . Deep value-based methods like DDQN ( van Hasselt et al. 2016 ) and TD3 ( Fujimoto et al. 2018 ) approximate Q with a neural network and learn from transitions (s,a,r,s^{\prime}) sampled from a replay buffer via temporal-difference (TD) updates. Reward hacking. Let R denote the observed training reward and R^{*} the intended reward (unobserved by the agent). The agent’s true objective is to maximize J_{R^{*}}(\pi) , while observing only rewards from R . For a policy update from \pi to \pi^{\prime} , we say the update induces reward hacking if J_{R}(\pi^{\prime})>J_{R}(\pi) but J_{R^{*}}(\pi^{\prime})<J_{R^{*}}(\pi) . In words, the update looks better under the proxy while making intended performance worse. Skalse et al. 2022 define hackability as a property of reward-function pairs; our notion focuses on the policy update and is complementary. 3 Method Modification-Considering Value Learning (MCVL) wraps an off-policy learner and treats each learning update as a candidate modification to be evaluated before adoption. Because the desired objective is not observed, MCVL uses a learned current return estimator as a proxy to accept or reject updates. MCVL poses a counterfactual question: if it were to allocate the next l training steps either (i) to its current replay buffer \mathcal{D} alone or (ii) to \mathcal{D} augmented with the new transition T_{\mathrm{new}} , which resulting policy would achieve a higher expected return according to the agent’s current bootstrapped-return estimator ? Both branches use the same compute budget and are scored by the same evaluator. The transition is accepted if and only if adding T_{\mathrm{new}} does not decrease the score. This yields a locally rational accept/reject rule under the agent’s present preferences. Current bootstrapped-return estimator. MCVL maintains a reward model R_{\psi}(s,a) trained by supervised regression on observed rewards and an action-value function Q_{\theta}(s,a) trained with standard TD targets. Together they define a current n -step bootstrapped return estimator for a trajectory \tau=(s_{0},a_{0},\ldots,s_{n-1},a_{n-1},s_{n},a_{n}) : \hat{G}_{n}(\tau)\;=\;\sum_{t=0}^{n-1}\gamma^{t}\,R_{\psi}(s_{t},a_{t})\;+\;\gamma^{n}\,Q_{\theta}(s_{n},a_{n}). (1) During scoring, the estimator parameters (\psi,\theta) are frozen to the live agent’s current values . Policy forecasting and comparison. Upon observing T_{\mathrm{new}}=(s,a,r,s^{\prime}) , MCVL constructs two forecasts under an identical training budget of l learner updates: \tilde{\pi}^{\,0}\;=\;\mathsf{Forecast}(\mathcal{D},\,l),\qquad\tilde{\pi}^{\,+}\;=\;\mathsf{Forecast}(\mathcal{D}\cup\{T_{\mathrm{new}}\},\,l). The operator \mathsf{Forecast}(\cdot,l) clones the current networks and runs l base-learner updates on minibatches from the specified dataset. These updates do not affect the live agent. Both forecasts are scored by the same frozen evaluator from Equation 1 . Let \{s^{(i)}\}_{i=1}^{k}\sim\rho be a set of initial states and let rollouts be of length n under the same transition model \hat{P} for both branches. Define \hat{J}(\pi)\;=\;\frac{1}{k}\sum_{i=1}^{k}\;\big[\hat{G}_{n}(\tau\sim(\hat{P},\pi)\,|\,s_{0}=s^{(i)})\big]. (2) as the approximated expected on-policy return under the current bootstrapped return estimator. It is used for evaluating T_{\mathrm{new}} : MCVL admits T_{\mathrm{new}} if and only if \hat{J}(\tilde{\pi}^{\,+})\geq\hat{J}(\tilde{\pi}^{\,0}) . Using matched compute, evaluator parameters, and transition model isolates the effect of admitting T_{\mathrm{new}} and makes the comparison less sensitive