Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward

Xuexiang Wen, Hang Yu, Linchao Zhu, Gaoang Wang · 2026

Read Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward by Xuexiang Wen, Hang Yu, Linchao Zhu, Gaoang Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09920.txt.

Description

Wen and three colleagues proposed VIGOR, a verifier-free reinforcement-learning reward built only from the policy model itself. Reinforcement learning with verifiable rewards needs gold labels or domain verifiers, which limits scaling to new tasks. VIGOR samples a group of completions per prompt and rewards outputs with smaller gradient norms of teacher-forced negative log-likelihood under current parameters.

Smaller gradient norms suggest a completion aligns better with the current policy, giving an intrinsic preference signal. A square-root-of-T scaling corrects the length bias of averaged token gradients, and group-wise rank shaping steadies reward scales. On math reasoning benchmarks, VIGOR beat the RLIF baseline and transferred to code when trained only on math. On Qwen2.5-7B-Base post-trained on MATH, it gained 3.31 points in math accuracy and 1.91 points in code accuracy over that baseline.

Catalog record

CREATORXuexiang Wen, Hang Yu, Linchao Zhu, Gaoang Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09920

Browse the full library · About the Tower of Babel