Regularized Online RLHF with Generalized Bilinear Preferences
Junghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun, Se-Young Yun · 2026
Read Regularized Online RLHF with Generalized Bilinear Preferences by Junghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun, Se-Young Yun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-23116.txt.
Description
Junghyun Lee and four co-authors prove fast penalized RLHF under general bilinear tastes. They study penalized best-response regret cutting in online reward learning from human feedback under general tastes and bandit feedback. While various penalties are used to toughen matching, known polylog regret promises remain mostly tied to KL divergence, raising the question of whether fast rates extend beyond it.
Adopting the General Bilinear Taste Model, which catches cyclical tastes over item-wise features via a low-rank skew matrix, the authors separate the impact of general penalties. Key: under GBPM, the dual gap of any greedy policy is bounded by the squared guessing error, shown using only strong bend and skew. With feature cover, Greedy Sampling gives general polylog regret. Explore-Then-Commit gives a per-dimension improved regret for well-set arm sets. Fast regrets thus follow from general strongly bent shape rather than KL alone.
Catalog record
| CREATOR | Junghyun Lee, Minju Hong, Kwang-Sung Jun, Chulhee Yun, Se-Young Yun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 131 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-23116