Clipping-Free Policy Optimization for Large Language Models
Ömer Veysel Çağatan, Barış Akgün, Gözde Gül Şahin, Xuandong Zhao · 2026
Read Clipping-Free Policy Optimization for Large Language Models by Ömer Veysel Çağatan, Barış Akgün, Gözde Gül Şahin, Xuandong Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22801.txt.
Description
Ömer Veysel Çağatan and three co-authors propose a new way to train large language models with rewards. The standard methods clip, or cap, each update to keep training stable. At large scale, clipping can stall learning, invite reward hacking, and make training shaky.
Their method, CFPO, drops clipping and uses a smooth penalty based on total variation distance instead. The penalty has no hard edges, so updates stay stable but never fully flat. They test it on reasoning and alignment tasks, two core uses of reward-based training.
Catalog record
| CREATOR | Ömer Veysel Çağatan, Barış Akgün, Gözde Gül Şahin, Xuandong Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22801