Reward Shaping for Inference-Time Alignment: A Stackelberg Game Perspective

Haichuan Wang, Tao Lin, Lingkai Kong, Hezi Jiang, Milind Tambe · 2026

Read Reward Shaping for Inference-Time Alignment: A Stackelberg Game Perspective by Haichuan Wang, Tao Lin, Lingkai Kong, Hezi Jiang, Milind Tambe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02572.txt.

Description

Haichuan Wang and 5 co-authors redesign the reward model itself to get more user utility out of aligned language models, in a 2026 arXiv paper. Standard alignment optimizes a policy against a learned reward with a penalty for straying from the base model. That penalty can make the model inherit base-policy biases that conflict with user preferences. Amplifying rewards helps but invites reward hacking.

The authors formalize reward design as a Stackelberg game, where a leader sets the reward and the policy follows. A simple reward shaping scheme approximates the optimal reward model well. Evaluated in inference-time alignment settings, the method plugs into existing pipelines with minimal overhead. Average reward improved consistently across settings. Win-tie rates against every baseline exceeded sixty-six percent on average. Better rewards, not just better optimization, unlock preference gains.

Catalog record

CREATORHaichuan Wang, Tao Lin, Lingkai Kong, Hezi Jiang, Milind Tambe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02572

Browse the full library · About the Tower of Babel