Regularized Online RLHF with Generalized Bilinear Preferences

Junghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun, Se-Young Yun · 2026

Read Regularized Online RLHF with Generalized Bilinear Preferences by Junghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun, Se-Young Yun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23116.txt.

Description

Junghyun Lee and four co-authors prove fast penalized RLHF under general bilinear tastes. They study penalized best-response regret cutting in online reward learning from human feedback under general tastes and bandit feedback. While various penalties are used to toughen matching, known polylog regret promises remain mostly tied to KL divergence, raising the question of whether fast rates extend beyond it.

Adopting the General Bilinear Taste Model, which catches cyclical tastes over item-wise features via a low-rank skew matrix, the authors separate the impact of general penalties. Key: under GBPM, the dual gap of any greedy policy is bounded by the squared guessing error, shown using only strong bend and skew. With feature cover, Greedy Sampling gives general polylog regret. Explore-Then-Commit gives a per-dimension improved regret for well-set arm sets. Fast regrets thus follow from general strongly bent shape rather than KL alone.

Catalog record

CREATORJunghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun, Se-Young Yun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE131 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-23116

Browse the full library · About the Tower of Babel