Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
Kenton Tang, Yuzhu Chen, Fengxiang He · 2026
Read Generalisation of RLHF under Reward Shift and Clipped KL Regularisation by Kenton Tang, Yuzhu Chen, Fengxiang He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21765.txt.
Description
Alignment and adaptation in language models lean on reinforcement learning from human feedback. Theory for its generalization remains premature, especially when the reward can shift and the KL control is estimated and clipped. Kenton Tang and two co-authors build generalization theory for RLHF covering two issues. Reward shift means reward models train on preference data from earlier or mixed policies while RLHF optimizes the policy on its rollouts. Clipped KL regularization means the KL term is estimated from sampled log-probability ratios, clipped for stability.
Their generalization bounds trace error to sampling error from prompts and rollouts, reward shift error, and KL clipping error. Special cases cover initializing with a uniform prior over a finite space and training by stochastic gradient descent as an Ornstein-Uhlenbeck process. The theory yields guidance on the optimal KL clipping threshold and on budget allocation across prompts, rollouts, and preference data.
Catalog record
| CREATOR | Kenton Tang, Yuzhu Chen, Fengxiang He |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 95 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-21765