Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models

Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary · 2026

Read Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models by Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30627.txt.

Introduction

Subramanyam Sahoo and three colleagues report in 2026 that a safety habit can backfire. Conservative offline training keeps a model's policy close to well-supported behaviour, and is widely trusted as a safe base for later online adaptation. The intuition says such a model should exploit flaws in a learned reward model less, not more. The team tested this belief directly on a reasoning model.

They trained a Qwen3 policy with direct preference optimisation at three levels of conservatism, then adapted each version online against a learned reward ensemble. True skill was measured by exact-answer accuracy on grade-school maths problems. More conservatism steadily increased reward-hacking damage, with a perfect rank correlation across the three settings. The mechanism ran through falling response diversity: narrow outputs clustered where the reward model was trained, yet ensemble disagreement there grew and was exploited faster. The authors recommend calibrated, not maximal, conservatism.

Catalog record

CREATORSubramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30627

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary Abstract Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model. We challenge this intuition empirically and mechanistically. We train a Qwen3-14B policy under Direct Preference Optimisation (DPO) with three levels of conservatism ( β ∈ { β lo , β mid , β hi } \beta\in\{\beta_{\mathrm{lo}},\beta_{\mathrm{mid}},\beta_{\mathrm{hi}}\} derived from empirical log-ratio percentiles), then adapt each checkpoint online against a learned reward ensemble (3 × \times Qwen3-1.7B) while measuring true performance on GSM8K exact-answer accuracy. We find that higher offline conservatism monotonically increases reward-hacking damage , measured by the Goodhart gap and its area under the curve (AUGC), with Spearman ρ = 1.0 \rho=1.0 across all three conditions. Mechanistic analysis reveals a three-link causal chain: (i) high- β \beta DPO compresses policy entropy, (ii) Low-entropy policies generate responses with reduced diversity, concentrating in a narrow region of the reward model’s training distribution (lower pairwise cosine distance), and (iii) despite this proximity, ensemble disagreement (epistemic uncertainty) increases with β \beta and is exploited faster during online optimisation. We further fit a power-law curve to the ( β , AUGC ) (\beta,\mathrm{AUGC}) data and identify a practical optimal conservatism level β ⋆ \beta^{\star} that balances alignment fidelity against hacking vulnerability. Our results suggest that the field needs calibrated , not maximal , conservatism. † † footnotetext: Code: Conservative-Offline-Training-Amplifies-Reward-Hacking-During-Online-Adaptation 1 Introduction The standard recipe for safe language model alignment runs as follows: first perform offline training on human-preference data (e.g., via RLHF ( 3 ) or DPO ( 16 ) ) using a conservatism coefficient \beta that penalises deviation from a reference policy, then optionally adapt online using a learned proxy reward. The implicit contract is that a more conservative offline checkpoint enters the online phase closer to the distribution where the reward model was trained, and therefore exploits it less aggressively. The Paradox We demonstrate empirically and mechanistically that the opposite can happen: higher offline conservatism ( \beta ) leads to greater reward-hacking damage during online adaptation . The mechanism is not arbitrary; it arises from a principled chain of events rooted in entropy compression and out-of-distribution extrapolation. To make this concrete, consider what a high- \beta DPO objective actually does to the policy. It tightens the KL constraint against \pi_{\mathrm{ref}} , concentrating probability mass on a narrow slice of token sequences that \pi_{\mathrm{ref}} already assigns high density. The resulting policy has low output entropy and low response diversity. When this compressed policy is then optimised against a learned reward ensemble, two effects combine. First, the reward model was trained on a relatively diverse set of human-preference pairs; the compressed policy generates responses that lie in sparse regions of that training distribution, causing high epistemic uncertainty (ensemble disagreement). Second, a low-entropy starting point has fewer exploration directions through which gradient updates can improve true performance, so the optimiser rapidly channels all gradient signal toward reward-model blind spots. The net result is that the Goodhart gap—the divergence between proxy and true reward—opens faster and wider for high- \beta policies. This paper makes four contributions. 1. Empirical demonstration of the paradox using real models (Qwen3-14B policy, Qwen3-1.7B reward ensemble) and real data (UltraFeedback, GSM8K). 2. Mechanistic attribution via response-entropy collapse measurements and reward-model OOD-distance analysis. 3. A power-law fit of AUGC as a function of \beta , yielding a practical design principle: an optimal \beta^{\star} exists below which online safety degrades faster than offline alignment improves. 4. Algorithmic and benchmark recommendations for the next generation of conservative-alignment methods that explicitly trade off pessimism against hacking vulnerability. 2 Background and Related Work 2.1 Direct Preference Optimisation DPO ( 16 ) sidesteps the need for an explicit reward model by reparameterising the RLHF objective directly in terms of policy log-ratios. Given a preference dataset \mathcal{D}=\{(x,y_{w},y_{l})\} of prompts with winning and losing responses, DPO minimises \mathcal{L}_{\mathrm{DPO}}(\pi_{\theta};\pi_{\mathrm{ref}})=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\Big[\log\sigma\Big(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\mathrm{ref}}(y_{w}\mid x)}\\ -\beta\log\frac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\mathrm{ref}}(y_{l}\mid x)}\Big)\Big]. (1) where \sigma is the logistic function and \beta>0 is the conservatism coefficient. Larger \beta imposes a tighter implicit KL constraint \KL(\pi_{\theta}\|\pi_{\mathrm{ref}}) , pulling the learned policy closer to the reference. 2.2 Goodhart’s Law in RLHF Goodhart’s Law ( 8 ) states that when a measure becomes a target, it ceases to be a good measure. In the RLHF context, 7 formalise this as a monotone degradation of true performance as the policy drifts toward maximising a proxy reward. 17 give a taxonomy of reward hacking and show it is near-inevitable when the reward model has any imperfection. Our work studies a less-explored facet: how the offline training strategy modulates the severity and speed of hacking during online adaptation. 2.3 Reward Model Ensembles and Uncertainty Ensembling is the standard approach for epistemic uncertainty estimation in deep neural networks ( 13 ) . In the reward-model context, ensemble disagreement provides a proxy for out-of-distribution inputs and has been used as a pessimism signal in offline RL ( 12 ; 10 ) . Our ensemble (3 \times Qwen3-1.7B trained with bootstrap resampling) provides both the proxy reward signal and the uncertainty signal used in our mechanistic analysis. 2.4 Offline RL and Conservative Methods The connection between DPO and offline RL is well known. CQL ( 12 ) penalises Q-values for out-of-support actions; IQL ( 11 ) avoids OOD actions by replacing Bellman backups with implicit quantile regression. The Decision Transformer ( 2 ) reframes RL as sequence modelling, which is directly analogous to DPO. All of these methods share the same conservatism logic encoded in Equation 1 : larger conservatism coefficient \to policy stays closer to reference \to (assumed) safer online performance. We show this assumption is violated under a specific failure mode: OOD-driven reward hacking. 3 Problem Formulation Let \pi_{\theta} denote the policy being trained, \pi_{\mathrm{ref}} the frozen DPO checkpoint (which itself used \pi_{\mathrm{ref}}^{(0)} as its reference), and \hat{r}_{\phi}(x,y) a learned proxy reward parameterised by an ensemble \phi=\{\phi_{k}\}_{k=1}^{K} . The true reward r^{\star}(x,y) is verifiable (GSM8K exact-answer accuracy) and is observed only during evaluation, never during training. Definition 3.1 (Goodhart Gap) . At online step t , define the batch-averaged normalised rewards \tilde{r}_{\mathrm{proxy}}(t)=\frac{\bar{r}_{\mathrm{proxy}}(t)}{\lvert\bar{r}_{\mathrm{proxy}}(0)\rvert+\varepsilon},\quad\tilde{r}_{\mathrm{true}}(t)=\frac{\bar{r}_{\mathrm{true}}(t)}{\lvert\bar{r}_{\mathrm{true}}(0)\rvert+\varepsilon}, where \varepsilon=\epsilon_{\mathrm{float32}} . The Goodhart gap is \mathcal{G}(t;\beta)=\tilde{r}_{\mathrm{proxy}}(t)-\tilde{r}_{\mathrm{true}}(t). (2) Definition 3.2 (AUGC) . The area under the Goodhart gap curve (AUGC) measures cumulative hacking damage over the online run: \mathrm{AUGC}(\beta)=\int_{0}^{T}\max\!\bigl(\mathcal{G}(t;\beta),\,0\bigr)\,\mathrm{d}t. (3) The central question of this paper is: does \mathrm{AUGC}(\beta) increase or decrease with \beta ? Conventional wisdom Offline RL theory predicts \mathrm{AUGC}(\beta) should decrease with \beta : a more conservative policy stays closer to \pi_{\mathrm{ref}} , which lies within the reward model’s training support, so it should hack less. Empirical finding We observe the opposite: \mathrm{AUGC}(\beta) increases monotonically with \beta (Spearman \rho=1.0 , p<0.05 ). 4 Experimental Setup 4.1 Models and Data Policy model. We use Qwen/Qwen3-14B as the policy, loaded in 4-bit NF4 QLoRA quantisation ( 6 ) with LoRA adapters ( 9 ) applied to all attention projection matrices. The LoRA rank r is derived architecturally: r=2^{\lfloor\log_{2}\sqrt{h_{\mathrm{hidden}}}\rceil},\quad\alpha=2r, (4) where \lceil\cdot\rceil denotes rounding to nearest integer and h_{\mathrm{hidden}} is the model hidden dimension. The

Read the complete text