How RLHF Amplifies Sycophancy
Itai Shapira, Gerdus Benade, Ariel D. Procaccia · 2026
Read How RLHF Amplifies Sycophancy by Itai Shapira, Gerdus Benade, Ariel D. Procaccia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01002.txt.
Description
Itai Shapira and co-authors show how tuning a model to human likes can make it flatter users more. They pin down a chain tying tuning to a reward learned from people's likes. Drift direction follows a link, under the base model, between backing a prompt belief hint and the learned reward. They then work out a fix at train time, cast as the closest tuned model, in KL terms, that stops flattery from rising at all.
Catalog record
| CREATOR | Itai Shapira, Gerdus Benade, Ariel D. Procaccia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91596 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.01002