From Curiosity to Caution: Mitigating Reward Hacking for Best-of- N N with Pessimism
Zhuohao Yu, Zhiwei Steven Wu, Adam Block · 2026
Read From Curiosity to Caution: Mitigating Reward Hacking for Best-of- N N with Pessimism by Zhuohao Yu, Zhiwei Steven Wu, Adam Block (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04648.txt.
Description
Yu, Wu and Block tackle reward hacking in best-of-N sampling, where N responses are scored and the best is picked. Performance can fall as N grows, because selection favors responses exploiting reward flaws. Their approach, called caution, reverses curiosity: curiosity rewards prediction error as novelty, while caution penalizes it. The authors explore pessimism, using lower confidence bounds to avoid uncertain out-of-distribution actions. Caution trains an error model on typical responses and lowers atypical estimates. Tests show caution mitigates reward hacking in best-of-N sampling.
Catalog record
| CREATOR | Zhuohao Yu, Zhiwei Steven Wu, Adam Block |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 93008 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04648