On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR

Hao Ye Jisheng Dang Junfeng Fang Bimei Wang Yizhou Zhang Ning Lv, Wencan Zhang Hong Peng Bin Hu Tat-Seng Chua · 2026

Read On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR by Hao Ye Jisheng Dang Junfeng Fang Bimei Wang Yizhou Zhang Ning Lv, Wencan Zhang Hong Peng Bin Hu Tat-Seng Chua (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06523.txt.

Description

Hao Ye and nine co-authors study how reinforcement learning with verifiable rewards reshapes a model, focusing on low-rank parts of its weights. Earlier work found the new reasoning skill sits mainly in rank-1 components. Using periodic rank-1 substitution, the team spots implicit reward overfitting: test performance can look fine even while training rewards stay low.

They report three further patterns. The rank-1 component keeps little knowledge beyond math reasoning, training optimizes a specific singular spectrum with heavy-tailed values, and the linked singular vectors align more strongly over time. Together the findings suggest this training mainly improves sampling efficiency and may guide continual learning methods.

Catalog record

CREATORHao Ye Jisheng Dang Junfeng Fang Bimei Wang Yizhou Zhang Ning Lv, Wencan Zhang Hong Peng Bin Hu Tat-Seng Chua
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06523

Browse the full library · About the Tower of Babel