Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification
Jiahui Li *, Jianfeng Shan *, Wenpei Chen, Shunyu Wu, Jian Lou, Wenjie Feng, Dan Li, See-Kiong Ng · 2026
Read Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification by Jiahui Li *, Jianfeng Shan *, Wenpei Chen, Shunyu Wu, Jian Lou, Wenjie Feng, Dan Li, See-Kiong Ng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03608.txt.
Description
Li and co-authors improve label-free reinforcement learning for reasoning models at test time. Test-time reinforcement learning trains without labels, using the model's own outputs. Past work chased Pass at 1, but Pass at k coverage matters too. Direct Pass at k designs from supervised settings perform poorly here. The team diagnoses two root causes of the failure.
Low-confidence samples get pseudo-labels that are often wrong. High-confidence samples collapse into narrow answer diversity. Their framework, TTRL-CoCoV, adapts verification by confidence. High-confidence samples get an exploration reward to keep diversity. Low-confidence samples let a verifier pick the pseudo-label. Medium-confidence samples skip verification entirely. Six benchmarks show gains over the best rivals. Pass at 1 rises by about 10 percent on average, and Pass at 16 by about 19 percent. Even fully supervised RL methods are beaten on some benchmarks.
Catalog record
| CREATOR | Jiahui Li *, Jianfeng Shan *, Wenpei Chen, Shunyu Wu, Jian Lou, Wenjie Feng, Dan Li, See-Kiong Ng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03608