Measuring Human Preferences in RLHF is a Social Science Problem

Bijean Ghafouri, Eun Cheol Choi, Priyanka Dey, Emilio Ferrara · 2026

Read Measuring Human Preferences in RLHF is a Social Science Problem by Bijean Ghafouri, Eun Cheol Choi, Priyanka Dey, Emilio Ferrara (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03238.txt.

Introduction

Ghafouri and colleagues argue that reinforcement learning from human feedback rests on a shaky premise: annotator responses often are not genuine preferences at all. People may answer without opinions, build preferences on the spot, or read the same question differently, and these flaws bite hardest on value judgments. Filtering inconsistent annotators flipped harm verdicts for about 19 percent of prompts, showing the field often models noise as signal.

Catalog record

CREATORBijean Ghafouri, Eun Cheol Choi, Priyanka Dey, Emilio Ferrara
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE139347
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03238

Browse the full library · About the Tower of Babel