Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems

Christian Walder & Deep Karkhanis · 2025

Read Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems by Christian Walder & Deep Karkhanis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15201.txt.

Description

Christian Walder and Deep Karkhanis retrain the target of reinforcement learning: instead of one good try, aim for one good try among k attempts. Standard training rewards each sampled solution on its own. That favors single strong samples and starves exploration.

Their method reshapes the final rewards so training directly optimizes the best-of-k score. They derive low-noise, unbiased estimators for that score and its gradient. The trick works for yes-no rewards and graded ones alike. In practice, standard training runs with jointly reshaped rewards, a stable swap. Hard problems, where most tries fail, gain the most. Sampling budgets already spent can finally count fully.

Catalog record

CREATORChristian Walder & Deep Karkhanis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15201

Browse the full library · About the Tower of Babel