Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards
Deokgyu Yoon, Hyungkyu Kang, Joongkyu Lee, Byeongchan Kim, Gyungin Shin, Sungrae Park, Min-hwan Oh, Upstage {hhbdgy,jklee0717,bckim97,minoh}@snu.ac.kr{hkang,noel,sungrae.park}@upstage.ai · 2026
Read Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards by Deokgyu Yoon, Hyungkyu Kang, Joongkyu Lee, Byeongchan Kim, Gyungin Shin, Sungrae Park, Min-hwan Oh, Upstage {hhbdgy,jklee0717,bckim97,minoh}@snu.ac.kr{hkang,noel,sungrae.park}@upstage.ai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20865.txt.
Description
Deokgyu Yoon and six colleagues propose multi-step likelihood-ratio correction for reinforcement learning with verifiable rewards. Widely used PPO surrogate goals are fundamentally local, relying on a local reading of the exact policy gradient goal. That approximation improves stability by cutting variance from importance sampling, a known trick. Yet it introduces structural bias that trust-region mechanisms must control. Their N-step forward trace augments the PPO surrogate using the summed likelihood ratio of the next N minus one tokens.
Catalog record
| CREATOR | Deokgyu Yoon, Hyungkyu Kang, Joongkyu Lee, Byeongchan Kim, Gyungin Shin, Sungrae Park, Min-hwan Oh, Upstage {hhbdgy,jklee0717,bckim97,minoh}@snu.ac.kr{hkang,noel,sungrae.park}@upstage.ai |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65887 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.20865