Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems
Christian Walder & Deep Karkhanis · 2025
Read Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems by Christian Walder & Deep Karkhanis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15201.txt.
Description
Christian Walder and Deep Karkhanis retrain the target of reinforcement learning: instead of one good try, aim for one good try among k attempts. Standard training rewards each sampled solution on its own. That favors single strong samples and starves exploration.
Their method reshapes the final rewards so training directly optimizes the best-of-k score. They derive low-noise, unbiased estimators for that score and its gradient. The trick works for yes-no rewards and graded ones alike. In practice, standard training runs with jointly reshaped rewards, a stable swap. Hard problems, where most tries fail, gain the most. Sampling budgets already spent can finally count fully.
Catalog record
| CREATOR | Christian Walder & Deep Karkhanis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15201