Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards

Deokgyu Yoon, Hyungkyu Kang, Joongkyu Lee, Byeongchan Kim, Gyungin Shin, Sungrae Park, Min-hwan Oh, Upstage {hhbdgy,jklee0717,bckim97,minoh}@snu.ac.kr{hkang,noel,sungrae.park}@upstage.ai · 2026

Read Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards by Deokgyu Yoon, Hyungkyu Kang, Joongkyu Lee, Byeongchan Kim, Gyungin Shin, Sungrae Park, Min-hwan Oh, Upstage {hhbdgy,jklee0717,bckim97,minoh}@snu.ac.kr{hkang,noel,sungrae.park}@upstage.ai (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-20865.txt.

Description

Deokgyu Yoon and six colleagues propose multi-step likelihood-ratio correction for reinforcement learning with verifiable rewards. Widely used PPO surrogate goals are fundamentally local, relying on a local reading of the exact policy gradient goal. That approximation improves stability by cutting variance from importance sampling, a known trick. Yet it introduces structural bias that trust-region mechanisms must control. Their N-step forward trace augments the PPO surrogate using the summed likelihood ratio of the next N minus one tokens.

Catalog record

CREATORDeokgyu Yoon, Hyungkyu Kang, Joongkyu Lee, Byeongchan Kim, Gyungin Shin, Sungrae Park, Min-hwan Oh, Upstage {hhbdgy,jklee0717,bckim97,minoh}@snu.ac.kr{hkang,noel,sungrae.park}@upstage.ai
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65887
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.20865

Browse the full library · About the Tower of Babel