AlphaPO: Reward Shape Matters for LLM Alignment

Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal †, Natesh Pillai, S. Sathiya Keerthi † · 2025

Read AlphaPO: Reward Shape Matters for LLM Alignment by Aman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal †, Natesh Pillai, S. Sathiya Keerthi † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03884.txt.

Description

Aman Gupta and a large team argue that the shape of the reward used to train large language models to please people matters a great deal. Their paper, from two thousand twenty-five, sets out a new method called AlphaPO.

Fresh methods skip a split reward stage and read the reward straight from the policy being trained. But they can suffer a flaw: the odds of the liked replies often fall when they should rise. Gupta's team says the flaw turns on the reward's shape. AlphaPO adds a dial that lets trainers bend that shape past the standard form. The aim is steadier training and replies that match what people chose.

Catalog record

CREATORAman Gupta, Shao Tang, Qingquan Song, Sirou Zhu, Jiwoo Hong, Ankan Saha, Viral Gupta, Noah Lee, Eunki Kim, Siyu Zhu, Parag Agrawal †, Natesh Pillai, S. Sathiya Keerthi †
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-03884

Browse the full library · About the Tower of Babel