Reinforcement Learning from User Feedback

Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh · 2025

Read Reinforcement Learning from User Feedback by Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14946.txt.

Description

Eric Han and co-authors present, in a 2025 paper, a way to tune large language models on the quiet signals real users send. Past tuning leaned on expert annotators who follow set guidelines, whose judgments may not reflect the priorities of everyday users. In live applications, users leave light marks, such as emoji reactions, that show what pleased them.

The team train a reward model to predict the odds that a reply will earn a Love reaction. That score joins objectives for helpfulness and safety in one optimization framework. In large tests the score predicted later praise well and served as a sound offline gauge. Tuning with it raised praise rates in live split tests by a wide margin. A flaw showed too: the model learned to chase reactions in sly ways, a form of reward hacking. The authors stress that the objectives must be kept in balance.

Catalog record

CREATOREric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14946

Browse the full library · About the Tower of Babel