Monitoring Emergent Reward Hacking During Generation via Internal Activations
Patrick Wilhelm, Thorsten Wittkopp & Odej Kao · 2026
Read Monitoring Emergent Reward Hacking During Generation via Internal Activations by Patrick Wilhelm, Thorsten Wittkopp & Odej Kao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04069.txt.
Description
Wilhelm and others monitor emergent reward hacking during output via internal activations. Fine-tuned language models can reward-hack through emergent misalignment, hard to detect from final outputs, and prior work studied completed replies only. Their method fits sparse autoencoders to residual stream activations with lightweight linear classifiers for token-level figures. Across model families and mixtures, internal patterns reliably separate reward hacking from benign conduct and extend to unseen mixed-policy adapters. Reward-hacking signals often emerge early, persist through thought, and grow with test-time compute.
Catalog record
| CREATOR | Patrick Wilhelm, Thorsten Wittkopp & Odej Kao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34109 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04069