Constrained Group Relative Policy Optimization

Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull · 2026

Read Constrained Group Relative Policy Optimization by Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05863.txt.

Description

Adding safety constraints to GRPO, the dominant critic free method for fine tuning language models, exposes a subtle failure in how rewards are combined. Roger Girgis, Rodrigue de Schaetzen, and 4 co-authors introduce Constrained GRPO, a Lagrangian based extension for constrained policy optimization. The standard practice scalarizes rewards before normalization, mixing reward and constraint signals into one number.

That order creates a coupling problem: because normalization is shared, changing one constraint multiplier also reshuffles the relative weight of the reward and the other constraints. The fix is simple but crucial: scalarize the standardized advantages instead of the raw rewards. The update becomes better conditioned, multiplier dynamics behave, and constraint enforcement stabilizes. Experiments span a controlled gridworld, a real autonomous driving benchmark, and a math reasoning task. Across all three, the constrained method adhered better to the stated constraints. The paper targets safety critical fine tuning, where constraints are the point.

Catalog record

CREATORRoger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05863

Browse the full library · About the Tower of Babel