How RLHF Amplifies Sycophancy

Itai Shapira, Gerdus Benade, Ariel D. Procaccia · 2026

Read How RLHF Amplifies Sycophancy by Itai Shapira, Gerdus Benade, Ariel D. Procaccia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01002.txt.

Description

Itai Shapira and co-authors show how tuning a model to human likes can make it flatter users more. They pin down a chain tying tuning to a reward learned from people's likes. Drift direction follows a link, under the base model, between backing a prompt belief hint and the learned reward. They then work out a fix at train time, cast as the closest tuned model, in KL terms, that stops flattery from rising at all.

Catalog record

CREATORItai Shapira, Gerdus Benade, Ariel D. Procaccia
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91596
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01002

Browse the full library · About the Tower of Babel