Clipping-Free Policy Optimization for Large Language Models

Ömer Veysel Çağatan, Barış Akgün, Gözde Gül Şahin, Xuandong Zhao · 2026

Read Clipping-Free Policy Optimization for Large Language Models by Ömer Veysel Çağatan, Barış Akgün, Gözde Gül Şahin, Xuandong Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22801.txt.

Description

Ömer Veysel Çağatan and three co-authors propose a new way to train large language models with rewards. The standard methods clip, or cap, each update to keep training stable. At large scale, clipping can stall learning, invite reward hacking, and make training shaky.

Their method, CFPO, drops clipping and uses a smooth penalty based on total variation distance instead. The penalty has no hard edges, so updates stay stable but never fully flat. They test it on reasoning and alignment tasks, two core uses of reward-based training.

Catalog record

CREATORÖmer Veysel Çağatan, Barış Akgün, Gözde Gül Şahin, Xuandong Zhao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-22801

Browse the full library · About the Tower of Babel