Rethinking Diverse Human Preference Learning through Principal Component Analysis

Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, Hanjie Chen · 2025

Read Rethinking Diverse Human Preference Learning through Principal Component Analysis by Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, Hanjie Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13131.txt.

Description

Feng Luo and co-authors rethink, in a 2025 paper, how models learn what people like, given that likes split in many ways. A single reward score tends to blur that range. Rich, fine-grained like data could help, but it costs much to gather. Their method breaks reward into parts with no such data. They take the gap, in vector form, between liked and disliked replies for many pairs. A classic split tool, PCA, then pulls out the main axes of that gap cloud.

Each axis acts as its own reward, for a trait such as helpfulness, safety, or humor. Staff can mix the axes to fit each user, with no new training. The scales stay easy to read, since each axis has a clear sense. Tests show the parts track real traits well. The frame suits personal AI that must fit many tastes.

Catalog record

CREATORFeng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, Hanjie Chen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13131

Browse the full library · About the Tower of Babel