From Curiosity to Caution: Mitigating Reward Hacking for Best-of- N N with Pessimism

Zhuohao Yu, Zhiwei Steven Wu, Adam Block · 2026

Read From Curiosity to Caution: Mitigating Reward Hacking for Best-of- N N with Pessimism by Zhuohao Yu, Zhiwei Steven Wu, Adam Block (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04648.txt.

Description

Yu, Wu and Block tackle reward hacking in best-of-N sampling, where N responses are scored and the best is picked. Performance can fall as N grows, because selection favors responses exploiting reward flaws. Their approach, called caution, reverses curiosity: curiosity rewards prediction error as novelty, while caution penalizes it. The authors explore pessimism, using lower confidence bounds to avoid uncertain out-of-distribution actions. Caution trains an error model on typical responses and lowers atypical estimates. Tests show caution mitigates reward hacking in best-of-N sampling.

Catalog record

CREATORZhuohao Yu, Zhiwei Steven Wu, Adam Block
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93008
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04648

Browse the full library · About the Tower of Babel