Constrained Group Relative Policy Optimization
Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull · 2026
Read Constrained Group Relative Policy Optimization by Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05863.txt.
Description
Adding safety constraints to GRPO, the dominant critic free method for fine tuning language models, exposes a subtle failure in how rewards are combined. Roger Girgis, Rodrigue de Schaetzen, and 4 co-authors introduce Constrained GRPO, a Lagrangian based extension for constrained policy optimization. The standard practice scalarizes rewards before normalization, mixing reward and constraint signals into one number.
That order creates a coupling problem: because normalization is shared, changing one constraint multiplier also reshuffles the relative weight of the reward and the other constraints. The fix is simple but crucial: scalarize the standardized advantages instead of the raw rewards. The update becomes better conditioned, multiplier dynamics behave, and constraint enforcement stabilizes. Experiments span a controlled gridworld, a real autonomous driving benchmark, and a math reasoning task. Across all three, the constrained method adhered better to the stated constraints. The paper targets safety critical fine tuning, where constraints are the point.
Catalog record
| CREATOR | Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05863