Reinforcement Learning from User Feedback
Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh · 2025
Read Reinforcement Learning from User Feedback by Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14946.txt.
Description
Eric Han and co-authors present, in a 2025 paper, a way to tune large language models on the quiet signals real users send. Past tuning leaned on expert annotators who follow set guidelines, whose judgments may not reflect the priorities of everyday users. In live applications, users leave light marks, such as emoji reactions, that show what pleased them.
The team train a reward model to predict the odds that a reply will earn a Love reaction. That score joins objectives for helpfulness and safety in one optimization framework. In large tests the score predicted later praise well and served as a sound offline gauge. Tuning with it raised praise rates in live split tests by a wide margin. A flaw showed too: the model learned to chase reactions in sly ways, a form of reward hacking. The authors stress that the objectives must be kept in balance.
Catalog record
| CREATOR | Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14946