A Descriptive and Normative Theory of Human Beliefs in RLHF
Sylee Dandekar, Shripad Deshmukh, Frank Chiu, W. Bradley Knox, Scott Niekum · 2025
Read A Descriptive and Normative Theory of Human Beliefs in RLHF by Sylee Dandekar, Shripad Deshmukh, Frank Chiu, W. Bradley Knox, Scott Niekum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01692.txt.
Description
Sylee Dandekar and co-authors argue that what labelers believe about an AI agent shapes the feedback used to train it, in this 2025 paper. Standard practice treats human preference as a readout of the person's reward values. Beliefs about the agent's skill may quietly bend those choices.
The team build a preference model with beliefs inside and bound the final policy error from belief mismatch. A human study confirmed that beliefs shift preferences and that simple prompts can move beliefs. Synthetic tests showed that assuming the agent is flawless often backfires. Matching beliefs to real skill, they argue, makes feedback training work better.
Catalog record
| CREATOR | Sylee Dandekar, Shripad Deshmukh, Frank Chiu, W. Bradley Knox, Scott Niekum |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01692