Reward Shaping for Inference-Time Alignment: A Stackelberg Game Perspective
Haichuan Wang, Tao Lin, Lingkai Kong, Hezi Jiang, Milind Tambe · 2026
Read Reward Shaping for Inference-Time Alignment: A Stackelberg Game Perspective by Haichuan Wang, Tao Lin, Lingkai Kong, Hezi Jiang, Milind Tambe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02572.txt.
Description
Haichuan Wang and 5 co-authors redesign the reward model itself to get more user utility out of aligned language models, in a 2026 arXiv paper. Standard alignment optimizes a policy against a learned reward with a penalty for straying from the base model. That penalty can make the model inherit base-policy biases that conflict with user preferences. Amplifying rewards helps but invites reward hacking.
The authors formalize reward design as a Stackelberg game, where a leader sets the reward and the policy follows. A simple reward shaping scheme approximates the optimal reward model well. Evaluated in inference-time alignment settings, the method plugs into existing pipelines with minimal overhead. Average reward improved consistently across settings. Win-tie rates against every baseline exceeded sixty-six percent on average. Better rewards, not just better optimization, unlock preference gains.
Catalog record
| CREATOR | Haichuan Wang, Tao Lin, Lingkai Kong, Hezi Jiang, Milind Tambe |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02572