On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR
Hao Ye Jisheng Dang Junfeng Fang Bimei Wang Yizhou Zhang Ning Lv, Wencan Zhang Hong Peng Bin Hu Tat-Seng Chua · 2026
Read On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR by Hao Ye Jisheng Dang Junfeng Fang Bimei Wang Yizhou Zhang Ning Lv, Wencan Zhang Hong Peng Bin Hu Tat-Seng Chua (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06523.txt.
Description
Hao Ye and nine co-authors study how reinforcement learning with verifiable rewards reshapes a model, focusing on low-rank parts of its weights. Earlier work found the new reasoning skill sits mainly in rank-1 components. Using periodic rank-1 substitution, the team spots implicit reward overfitting: test performance can look fine even while training rewards stay low.
They report three further patterns. The rank-1 component keeps little knowledge beyond math reasoning, training optimizes a specific singular spectrum with heavy-tailed values, and the linked singular vectors align more strongly over time. Together the findings suggest this training mainly improves sampling efficiency and may guide continual learning methods.
Catalog record
| CREATOR | Hao Ye Jisheng Dang Junfeng Fang Bimei Wang Yizhou Zhang Ning Lv, Wencan Zhang Hong Peng Bin Hu Tat-Seng Chua |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06523