Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning

Yu Luo, Shuo Han, Yihan Hu, Dong Li, Jianye Hao · 2026

Read Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning by Yu Luo, Shuo Han, Yihan Hu, Dong Li, Jianye Hao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03320.txt.

Description

Yu Luo and four co-authors revisit how reinforcement learning fine-tunes language models, and soften one of its harshest tools. Popular methods such as PPO and GRPO clip the policy ratio to stabilize training, but hard clipping throws away gradient signal from rare, high-return actions, the eureka moments of complex reasoning. It also makes slightly stale data unusable, wasting samples.

The authors show that constraining the variance of the policy ratio instead gives a principled, smooth relaxation of clipping. Their method, R2VPO, builds a primal-dual framework on that insight, stabilizing updates while preserving the signal from valuable trajectories. The aim is fine-tuning that keeps learning from its best surprises instead of truncating them, improving sample efficiency for reasoning models.

Catalog record

CREATORYu Luo, Shuo Han, Yihan Hu, Dong Li, Jianye Hao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03320

Browse the full library · About the Tower of Babel