AlphaPO: Reward Shape Matters for LLM Alignment
Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal †, Natesh Pillai, S. Sathiya Keerthi † · 2025
Read AlphaPO: Reward Shape Matters for LLM Alignment by Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal †, Natesh Pillai, S. Sathiya Keerthi † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03884.txt.
Description
Aman Gupta and a large team argue that the shape of the reward used to train large language models to please people matters a great deal. Their paper, from two thousand twenty-five, sets out a new method called AlphaPO.
Fresh methods skip a split reward stage and read the reward straight from the policy being trained. But they can suffer a flaw: the odds of the liked replies often fall when they should rise. Gupta's team says the flaw turns on the reward's shape. AlphaPO adds a dial that lets trainers bend that shape past the standard form. The aim is steadier training and replies that match what people chose.
Catalog record
| CREATOR | Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal †, Natesh Pillai, S. Sathiya Keerthi † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03884