Generalisation of RLHF under Reward Shift and Clipped KL Regularisation

Kenton Tang, Yuzhu Chen, Fengxiang He · 2026

Read Generalisation of RLHF under Reward Shift and Clipped KL Regularisation by Kenton Tang, Yuzhu Chen, Fengxiang He (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21765.txt.

Description

Alignment and adaptation in language models lean on reinforcement learning from human feedback. Theory for its generalization remains premature, especially when the reward can shift and the KL control is estimated and clipped. Kenton Tang and two co-authors build generalization theory for RLHF covering two issues. Reward shift means reward models train on preference data from earlier or mixed policies while RLHF optimizes the policy on its rollouts. Clipped KL regularization means the KL term is estimated from sampled log-probability ratios, clipped for stability.

Their generalization bounds trace error to sampling error from prompts and rollouts, reward shift error, and KL clipping error. Special cases cover initializing with a uniform prior over a finite space and training by stochastic gradient descent as an Ornstein-Uhlenbeck process. The theory yields guidance on the optimal KL clipping threshold and on budget allocation across prompts, rollouts, and preference data.

Catalog record

CREATORKenton Tang, Yuzhu Chen, Fengxiang He
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE95 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-21765

Browse the full library · About the Tower of Babel