Ratio-Variance Regularized Policy Optimization

Yu Luo, Shuo Han, Yihan Hu, Lei Lv, Huaping Liu, Fuchun Sun, Jianye Hao, Dong Li · 2026

Read Ratio-Variance Regularized Policy Optimization by Yu Luo, Shuo Han, Yihan Hu, Lei Lv, Huaping Liu, Fuchun Sun, Jianye Hao, Dong Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26784.txt.

Description

Luo and seven colleagues replace the hard clipping used in on-policy reinforcement learning with a softer constraint on policy ratio variance. Hard clipping truncates high-return, high-divergence updates wholesale. Their ratio-variance limit instead acts as a soft brake, keeping useful gradient signals and reusing stale off-policy data. Their R2VPO method, built on primal-dual optimization, beats PPO baselines on math reasoning and robotic control, with the largest gains on smaller models.

Catalog record

CREATORYu Luo, Shuo Han, Yihan Hu, Lei Lv, Huaping Liu, Fuchun Sun, Jianye Hao, Dong Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62414
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26784

Browse the full library · About the Tower of Babel