A Descriptive and Normative Theory of Human Beliefs in RLHF

Sylee Dandekar, Shripad Deshmukh, Frank Chiu, W. Bradley Knox, Scott Niekum · 2025

Read A Descriptive and Normative Theory of Human Beliefs in RLHF by Sylee Dandekar, Shripad Deshmukh, Frank Chiu, W. Bradley Knox, Scott Niekum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01692.txt.

Description

Sylee Dandekar and co-authors argue that what labelers believe about an AI agent shapes the feedback used to train it, in this 2025 paper. Standard practice treats human preference as a readout of the person's reward values. Beliefs about the agent's skill may quietly bend those choices.

The team build a preference model with beliefs inside and bound the final policy error from belief mismatch. A human study confirmed that beliefs shift preferences and that simple prompts can move beliefs. Synthetic tests showed that assuming the agent is flawless often backfires. Matching beliefs to real skill, they argue, makes feedback training work better.

Catalog record

CREATORSylee Dandekar, Shripad Deshmukh, Frank Chiu, W. Bradley Knox, Scott Niekum
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01692

Browse the full library · About the Tower of Babel