Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward
Xuexiang Wen, Hang Yu, Linchao Zhu, Gaoang Wang · 2026
Read Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward by Xuexiang Wen, Hang Yu, Linchao Zhu, Gaoang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09920.txt.
Description
Wen and three colleagues proposed VIGOR, a verifier-free reinforcement-learning reward built only from the policy model itself. Reinforcement learning with verifiable rewards needs gold labels or domain verifiers, which limits scaling to new tasks. VIGOR samples a group of completions per prompt and rewards outputs with smaller gradient norms of teacher-forced negative log-likelihood under current parameters.
Smaller gradient norms suggest a completion aligns better with the current policy, giving an intrinsic preference signal. A square-root-of-T scaling corrects the length bias of averaged token gradients, and group-wise rank shaping steadies reward scales. On math reasoning benchmarks, VIGOR beat the RLIF baseline and transferred to code when trained only on math. On Qwen2.5-7B-Base post-trained on MATH, it gained 3.31 points in math accuracy and 1.91 points in code accuracy over that baseline.
Catalog record
| CREATOR | Xuexiang Wen, Hang Yu, Linchao Zhu, Gaoang Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.09920