Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions
Simon Matrenok, Skander Moalla, Caglar Gulcehre · 2025
Read Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functions by Simon Matrenok, Skander Moalla, Caglar Gulcehre (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08068.txt.
Description
Simon Matrenok, Skander Moalla, and Caglar Gulcehre introduce a method for training language models from single-number rewards, in a 2025 paper. Aligning a model with such absolute rewards has needed online methods that sample as they train. Simpler offline methods could only learn from paired preferences or relative signals, which limits their data.
Their method uses quantile rewards, which let it aim straight at the known best answer for the training goal. That trick also gives an exact formula for a term that relative methods must cancel out. In chat and coding tests, the method matched or beat strong rivals at the 8-billion-parameter scale. Robust rewards also cut length bias.
Catalog record
| CREATOR | Simon Matrenok, Skander Moalla, Caglar Gulcehre |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 195 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-08068