Constrained Group Relative Policy Optimization Roger Girgis, Rodrigue de Schaetzen, Luke Rowe, Azalée Robitaille, Christopher Pal, Liam Paull Abstract While Group Relative Policy Optimization (GRPO) has emerged as a scalable framework for critic-free policy learning, extending it to settings with explicit behavioral constraints remains underexplored. We introduce Constrained GRPO, a Lagrangian-based extension of GRPO for constrained policy optimization. Constraints are specified via indicator cost functions, enabling direct optimization of violation rates through a Lagrangian relaxation. We show that a naive multi-component treatment in advantage estimation can break constrained learning: mismatched component-wise standard deviations distort the relative importance of the different objective terms, which in turn corrupts the Lagrangian signal and prevents meaningful constraint enforcement. We formally derive this effect to motivate our scalarized advantage construction that preserves the intended trade-off between reward and constraint terms. Experiments in a toy gridworld confirm the predicted optimization pathology and demonstrate that scalarizing advantages restores stable constraint control. In addition, we evaluate Constrained GRPO on robotics tasks, where it improves constraint satisfaction while increasing task success, establishing a simple and effective recipe for constrained policy optimization in embodied AI domains that increasingly rely on large multimodal foundation models. † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Overview of our proposed approach. We extend GRPO to constrained policy optimization by enforcing user-specified behavioral constraints with thresholds (left). Naively scalarizing reward distorts reward–constraint trade-offs, producing inconsistent effective weights (middle; e.g., red/green swap). Scalarized advantages preserves multiplier semantics, yielding stable learning and effective constraint enforcement (right). Recent progress in reinforcement learning (RL) has shown that large multimodal models can be optimized with outcome-based signals. This has enabled the finetuning of models on mathematical reasoning tasks ( 11 ; 49 ; 23 ) , embodied decision making ( 50 ) , among other domains where success can be evaluated with programmatic or learned rewards. Group Relative Policy Optimization (GRPO) ( 42 ) removes the reliance on a critic for advantage estimation by a within-group normalization of the rewards obtained over multiple sampled outputs for the same query. The critic-free structure of GRPO is appealing when training or deploying a value function is difficult. Existing GRPO-based methods have largely approached multi-objective optimization through manually weighting the importance of different reward terms ( 25 ; 19 ; 28 ) . In contrast, the constrained policy optimization literature offers a principled framework, namely Constrained Markov Decision Processes (CMPDs), for enforcing behavioural preferences and constraints by explicitly defining thresholds on behaviours rather than employing fixed weights. However, GRPO has not yet been integrated into a constrained optimization framework to yield reliable constraint enforcement. In this work, we propose Constrained GRPO , a constrained policy optimization framework employing GRPO as the policy optimization algorithm. Within our framework, we specify desirable behaviours with indicator cost functions ( 39 ) and enforce target behaviour rates (in percentage) using a Lagrangian relaxation with learned multipliers. A central challenge arises from the interaction between Lagrangian methods and GRPO’s advantage estimation through within-group standardization. A simple approach is to first scalarize rewards , i.e., combine reward and cost terms into a single scalar value using the current multipliers, and then apply the within-group standardization. However, we show that this seemingly innocuous choice can deteriorate the constrained optimization: the group standarization induces an implicit, covariance-dependent reweighting of the terms, distorting the intended relative importance prescribed by the Lagrange multipliers. We provide a mathematical analysis of this phenomenon and illustrate the resulting distortion in Figure 1 . Motivated by this analysis, Constrained GRPO employs scalarizing advantages , i.e., standardize each component within group first, and then combine the standardized components using the Lagrangian multipliers. By construction, this design preserves the intended prescription of the multipliers by preventing the within-group scales and correlations from implicitly reshaping the trade-off, a property we illustrate in our experiments. Prior and concurrent work has also found advantage scalarization beneficial in the realm of multi-objective GRPO ( 25 ; 19 ) ; our contribution places this observation in the context of constrained optimization, showing why it is essential for effective Lagrangian constraint enforcement under GRPO. Our main contributions are: (1) We introduce Constrained GRPO, a constrained policy optimization framework built on GRPO, using indicator cost functions, Lagrangian relaxation and scalarizing advantages to effectively enforce behaviour rates without manually tuning reward weights. (2) We analyze the common approach of scalarizing rewards before normalizing, and mathematically show in Theorem 4.1 that it implicitly reweighs the components as a function of within-group variances and correlations between terms, distorting the prescribed relative importance encoded by the Lagrangian multipliers. (3) We study the alternative approach of scalarizing advantages and empirically demonstrate improved stability and faithfulness of the behaviour specification in a gridworld environment and a realistic simulation-based autonomous driving setting. 2 Related Work 2.1 Reward Specification A central motivation for our work is the difficulty of specifying complex behaviours through a single reward function. This challenge has inspired several alternative approaches in the literature. Imitation Learning ( 53 ) bypasses reward engineering entirely by using expert demonstrations to define the task. Other methods introduce human feedback during training, either to guide the agent toward desired behaviours ( 9 ) or to prevent catastrophic errors during exploration ( 40 ) . Our approach shares a similar motivation—capturing designer intent more directly—but does so by specifying constraints via indicator functions and thresholds, avoiding the need for demonstrations or continuous human intervention. Another promising direction is the use of natural language to specify tasks ( 16 ; 29 ) . While this can provide a more intuitive interface for human designers, language is inherently ambiguous and can be exploited by agents (reward hacking). Moreover, these methods typically require training an additional language-to-reward model, adding further complexity. Another interesting approach is Inverse Reward Design ( 18 ; 30 ; 36 ) , which treats the provided reward function as a single observation of the designer’s latent intent and learns a probabilistic model over possible true reward functions. This is particularly useful in settings with changing environments, but our work focuses on fixed-distribution environments where explicit behaviour specification is more appropriate. 2.2 Multi-Objective Optimization with Large Models In the context of large multimodal (vision-)language models, there is extensive work on safety and multi-objective optimization via reinforcement learning ( 31 ; 51 ; 12 ; 21 ; 5 ) . For example, Constitutional AI ( 5 ) can be viewed as balancing multiple objectives—most notably helpfulness and harmlessness—yet in practice these objectives are typically reduced to a single scalar reward and optimized with standard RLHF ( 32 ) pipelines, for instance PPO-style updates ( 41 ) . 31 frame composite reward models in RLHF as a constrained optimization problem, enforcing thresholds on individual reward components via a CMDP formulation and using dynamic weight adjustment (i.e., Lagrange multipliers) to mitigate over-optimization. Similarly, 12 formulate alignment as maximizing reward subject to cost constraints in RLHF, explicitly trading off helpfulness against safety-related costs. Finally, 47 study multi-objective reward models in RLHF, emphasizing interpretability as a means to help designers verify adherence to human preferences. While several recent methods target safety alignment for RL-finetuned (vision-)language models, they are largely developed outside the GRPO setting. More recent work ( 25 ) train a multi-head reward model that predicts K{=}4 attributes (politeness, meaningfulness, actionability, safety) and then form a single scalar reward via a fixed weighted sum (using w_{k}{=}1 for all components), before optimizing the resulting objective with GRPO. 19 propose Multi-Objective GRPO (MO-GRPO) to reduce reward-hacking behavior under multiple objectives, and advocate scalarized advantages to equalize the influence of components with different variances. Concurrent to our work, GDPO ( 28 ) also supports advantage scalarization and emphasizes the difficulty of selecting reward weights that reflect desired priorities, proposing conditional objectives that activate certain terms only once prerequisite scores exceed a minimum threshold. In contrast, we study advantage scalarization through the lens of constrained policy optimization. Our goal is to ensure that increases in the Lagrangian multipliers in Constrained GRPO translate into stronger enforcement of the specified behaviors. As shown experimentally, scalarizing advantages provides this property, whereas scalarizing rewards can implicitly reweight objectives and undermine effective Lagrangian optimization. 3 Background Markov Decision Processes (MDPs). A Markov decision process (MDP) is defined by (\mathcal{S},\mathcal{A},P,R) ( 44 ) , where \mathcal{S} is the state space, \mathcal{A} the action space, P(\cdot\mid s,a) the transition kernel, and R(s,a) the reward function. At time t , the agent samples an action a_{t}\sim\pi(\cdot\mid s_{t}) , transitions according to s_{t+1}\sim P(\cdot\mid s_{t},a_{t}) , and receives reward R(s_{t},a_{t}) . The trajectory distribution under \pi and the expected discounted return are \displaystyle p_{\pi}(\tau) \displaystyle=P_{0}(s_{0})\prod_{t=0}^{T-1}P(s_{t+1}\mid s_{t},a_{t})\,\pi(a_{t}\mid s_{t}), \displaystyle J_{R}(\pi) \displaystyle=\mathbb{E}_{\tau\sim p_{\pi}}\Big[\sum_{t=0}^{T-1}\gamma^{t}R(s_{t},a_{t})\Big], where the objective is to learn a policy \pi^{*} that maximizes the expected discounted return, i.e. \pi^{*}=\arg\max_{\pi}J_{R}(\pi) . Constrained Markov Decision Processes (CMDPs). A constrained MDP (CMDP) augments the MDP objective with K expected cost constraints ( 2 ) . Given cost functions C_{k}:\mathcal{S}\times\mathcal{A}\to\mathbb{R} and thresholds d_{k} , define the feasible set of policies as \displaystyle\Pi_{C} \displaystyle=\Big\{\pi\in\Pi:\ J_{C_{k}}(\pi)\leq d_{k},\;\;k=1,\dots,K\Big\}, \displaystyle J_{C_{k}}(\pi) \displaystyle=\mathbb{E}_{\tau\sim p_{\pi}}\Big[\sum_{t=0}^{T-1}\gamma^{t}C_{k}(s_{t},a_{t})\Big]. The constrained optimal policy maximizes return subject to feasibility: \displaystyle\pi^{*}=\arg\max_{\pi\in\Pi}\ J_{R}(\pi)\quad\text{s.t.}\quad J_{C_{k}}(\pi)\leq d_{k},\;\;k=1,\dots,K, where stationary stochastic policies may be required for optimality ( 2 ) . Lagrangian methods. CMDPs are commonly addressed with Lagrangian methods ( 1 ; 37 ; 43 ; 52 ; 50 ) , which cast the problem as a saddle-point optimization ( 46 ; 35 ; 22 ) . These methods introduce nonnegative multipliers \lambda=\{\lambda_{k}\}_{k=1}^{K}\geq 0 , where each \lambda_{k} controls the penalty for constraint k and relative emphasis across constraints is determined by ratios such as \lambda_{i}/\lambda_{j} . The resulting min–max problem is \displaystyle\max_{\pi}\ \min_{\lambda\geq 0}\ \mathcal{L}(\pi,\lambda), (1) \displaystyle\text{where}\quad\mathcal{L}(\pi,\lambda) \displaystyle=J_{R}(\pi)-\sum_{k=1}^{K}\lambda_{k}\big(J_{C_{k}}(\pi)-d_{k}\big). A standard approach applies gradient ascent in \pi and gradient descent in \lambda ( 27 ) , analogous to training procedures used in GANs ( 15 ) . In practice, such schemes often attain good feasible solutions on CMDP benchmarks ( 1 ; 37 ; 43 ; 52 ) . Prior work ( 39 ) employed multiplier normalization: \lambda_{k}\;=\;\frac{\exp(z_{k})}{\exp(a_{R})+\sum_{k^{\prime}=1}^{K}\exp(z_{k^{\prime}})},\;\;k=1,\dots,K. which includes a dummy variable a_{R} for the main reward function such that \lambda_{R}:=1-\sum_{k=1}^{K}\lambda_{k} and z_{k} are the unnormalized multipliers. This mechanism ensures that one constraint can dominate the policy updates if necessary while maintaining stability. The policy update now corresponds to optimizing the linearly-combined reward signal, R_{c}(s,a)=\lambda_{R}R(s,a)-\sum_{k=1}^{K}\lambda_{k}\,C_{k}(s,a), (2) and can be carried out with any policy optimization method (e.g., PPO ( 41 ) , SAC ( 17 ) ). The multiplier update follows \displaystyle\nabla_{\lambda_{k}}\mathcal{L}(\pi,\lambda)=-(J_{C_{k}}(\pi)-d_{k}), (3) typically implemented with a learning rate and a projection onto \lambda_{k}\geq 0 using max-clipping. If constraint k is violated ( J_{C_{k}}>d_{k} ), \lambda_{k} increases, strengthening its penalty; if it is satisfied ( J_{C_{k}}0,\quad\sigma_{C_{k}}>0\;\;\forall k\in\{1,\dots,K\}, so that \displaystyle Z_{R} \displaystyle:=\frac{R-\mu_{R}}{\sigma_{R}}, (4) \displaystyle\quad Z_{C_{k}} \displaystyle:=\frac{C_{k}-\mu_{C_{k}}}{\sigma_{C_{k}}},\quad k=1,\dots,K, where \mu_{(\cdot)} and \sigma_{(\cdot)} denote the within-group mean and standard deviation of the corresponding component. Then the GRPO advantage obtained by standardizing the scalarized reward, A_{\mathrm{ScRew}}\;:=\;\frac{R_{S}-\mathbb{E}[R_{S}]}{\sigma_{R_{S}}}, admits the expansion A_{\mathrm{ScRew}}\;=\;\frac{(\mathbf{D}\boldsymbol{\lambda})^{\top}\mathbf{Z}}{\sigma_{R_{S}}}\;=\;\sum_{j=0}^{K}\Big(\frac{\lambda_{j}\,\sigma_{j}}{\sigma_{R_{S}}}\Big)\,Z_{j}, (5) where we index j=0 as the main reward component ( \lambda_{0}\!\equiv\!\lambda_{R} , \sigma_{0}\!\equiv\!\sigma_{R} ) and j=k corresponds to cost C_{k} . Consequently, the effective contribution of each standardized component Z_{j} is scaled by \lambda_{j}\sigma_{j}/\sigma_{R_{S}} , and therefore depends on within-group scales and (through \sigma_{R_{S}} ) the full covariance structure \mathbf{\Sigma} . The proof is provided in Appendix A . To summarize, our main observation is revealed in Eq. ( 5 ): under scalarized rewards, the effective contribution of each standardized component Z_{j} and the intended trade-offs encoded by the Lagrangian multipliers, \boldsymbol{\lambda} , are implicitly distorted by component variances and correlations. Scalarized Advantage. In our constrained GRPO approach, we first compute a GRPO-style standardized advantage per component (reward and each constraint) using the respective component’s within-group mean and standard deviation as done in Eq. ( 4 ). We then form a single advantage by linearly combining these standardized components using their current Lagrangian weights A_{\text{ScAdv}}\;=\;\lambda_{R}Z_{R}\;-\;\sum_{k=1}^{K}\lambda_{k}\,Z_{C_{k}}, (6) where \lambda_{k},\lambda_{R}\geq 0 are Lagrange multipliers associated with constraint k and the reward term, respectively. This yields an advantage signal whose relative trade-offs are explicitly controlled by the multipliers. 5 Illustrative Environment We begin with an illustrative example in a simple gridworld. An agent is randomly initialized in a 10\times 10 grid and must navigate to a randomly sampled goal location, receiving a sparse scalar reward upon reaching it. Each episode contains 20\% lava tiles; stepping onto lava incurs a penalty (i.e., a cost). In addition, the agent has a battery that depletes at every timestep and incurs a penalty whenever its battery level falls below 10\% . The action space consists of five discrete actions: moving in the four cardinal directions or staying in place to recharge. More details are available in Appendix C . Although intentionally minimal, this environment provides a controlled testbed for isolating the differences between scalarized rewards and scalarized advantages in unconstrained and constrained GRPO. We first compare these approaches under fixed reward and penalty weights. For simplicity, we restrict the battery-penalty weight to \lambda_{\text{battery}}^{\text{fixed}}\in\{0.0,0.1\} and sweep the lava-penalty weight over \lambda_{\text{lava}}^{\text{fixed}}\in\{0.0,0.01,0.05,0.1,0.2,0.3,0.5,0.7,1.0\} . Figure 2 illustrates how performance varies as we sweep \lambda_{\text{lava}}^{\text{fixed}} for the two fixed settings of \lambda_{\text{battery}}^{\text{fixed}} . Overall, we observe consistently higher goal-reaching rates with scalarized advantages , along with more stable performance across seeds (through smaller error bars). Moreover, scalarized rewards tends to unilaterally optimize the lava rate: even the smallest non-zero lava weight, \lambda_{\text{lava}}^{\text{fixed}}=0.01 , leads to near-minimal lava visitation. In contrast, scalarized advantages provides smoother, more gradual control over the lava rate, which we argue better reflects the intended meaning of the weights. When multiple penalty terms are active ( \lambda_{\text{battery}}^{\text{fixed}}=0.1 , i.e., bottom row in Fig. 2 ), scalarized advantages relaxes the lava rate for \lambda_{\text{lava}}^{\text{fixed}}\leq 0.5 , whereas scalarized rewards behaves identically to the \lambda_{\text{battery}}^{\text{fixed}}=0.0 setting, effectively optimizing the lava penalty in the same way regardless of the battery penalty. Figure 2 : Final performance of GRPO policies evaluated over 1,000 episodes in the gridworld with fixed penalty weights, comparing scalarized rewards (blue) and scalarized advantages (red). Scalarized advantages consistently achieves higher goal-reaching rates while tracking the intended trade-off induced by the lava-weight sweep more faithfully, whereas scalarized rewards suppresses lava visitation disproportionately. Next, we compare the scalarization strategies in a constrained setting. In this experiment, we enforce constraint thresholds \tilde{d}_{\text{lava}}=\tilde{d}_{\text{battery}}=0.01 and train for 512 k episodes. In Appendix C.3 , we show experiments with different threshold values. Figure 3 compares Constrained GRPO with scalarized advantages (ours) against a scalarized rewards baseline. Consistent with the fixed-weight results, scalarized advantages yields more stable learning across seeds (5 seeds), as reflected by the narrower shaded region throughout training (row 1). Moreover, scalarized advantages leads to richer behavior-rate dynamics (row 2, right column): the Lagrangian multipliers (row 3, right column) adapt to allow the constraint rates to increase toward the threshold, effectively encouraging the policy to use its available cost budget. On the other hand, scalarized rewards rapidly drives both constraint rates close to zero early in training (row 2, left column), and keeps them near-zero thereafter, even as the learned multipliers (row 3, left column) remain small. This behavior can be understood through the implicit reweighting predicted by Theorem 4.1 , and we refer to the effective scaling factors for the Z_{j} ’s in Eqs. ( 5 ) and ( 6 ) as effective weights . Observe that despite the small learned multipliers \lambda_{\text{lava}} and \lambda_{\text{battery}} , the induced effective weights are larger and significantly noisier due to within-group variance and covariance effects (row 4, left column). This is especially problematic when the ordering of importance changes, as observed around 150K steps for the goal and lava weights (red/green swap). Overall, these results support scalarized advantages as a more reliable advantage construction for Constrained GRPO, as it preserves the intended role of the Lagrangian multipliers in enforcing behavior specifications. Figure 3 : Learning dynamics on the gridworld under Constrained GRPO, comparing scalarized rewards (left) with scalarized advantages (right). Scalarized advantages yields more stable training and uses the available cost budget, with behavior rates adapting toward the specified thresholds as the Lagrangian multipliers adjust. In contrast, scalarized rewards collapses constraint rates toward near-zero early in training with the effective weights remaining larger and noisier despite small learned multipliers. 6 Robotics Environment 6.1 Benchmark Details Configuration Name NC \uparrow DAC \uparrow DDC \uparrow TLC \uparrow Ego Progress \uparrow Comfort \uparrow EPDMS \uparrow ADE@4s (m) \downarrow PDM-Closed (privileged) ( 13 ) 0.944 0.988 1.00 0.995 1.00 0.877 - - SFT Checkpoint 0.953 0.789 0.978 0.993 0.836 0.976 0.651 2.539 GRPO with EPDMS ( 20 ) 0.897 0.800 0.947 0.990 0.944 0.978 0.612 6.900 GRPO, ScRew, Equal Weights ( 25 ) 0.891 0.625 0.876 0.989 0.968 0.976 0.445 9.050 GRPO, ScAdv, Equal Weights ( 19 ) 0.908 0.661 0.900 0.989 0.954 0.973 0.490 7.172 GRPO, ScRew, Informed Weights 0.887 0.643 0.887 0.987 0.967 0.978 0.463 8.509 GRPO, ScAdv, Informed Weights 0.932 0.765 0.971 0.993 0.890 0.976 0.623 3.495 Constrained GRPO, ScRew 0.955 0.887 0.987 0.993 0.868 0.976 0.743 3.344 Constrained GRPO, ScAdv (ours) 0.957 0.928 0.989 0.998 0.796 0.977 0.774 3.361 Table 1 : Stage 1 results on the NAVSIM-v2 Navhard split: We compare the different baseline reward function configurations and our proposed Constrained GRPO approach. ScRew stands for Scalarized rewards, while ScAdv stands for Scalarized Advantages. We can see that the best performing models are the constrained approaches, with our the ScAdv variant performing best on EPDMS. We focus our experiments on the NAVSIM-v2 driving benchmark ( 7 ) , which contains 103 k training scenarios (Navtrain) and 12 k testing scenarios (Navtest) curated from OpenScene ( 34 ) , a redistribution of the nuPlan dataset ( 6 ) . NAVSIM-v2 is well-suited for this study because it evaluates planning using explicit behavioural safety constraints alongside task-level objectives such as progress and comfort, enabling targeted analysis of behaviour specification. It focuses on the most challenging driving scenarios from the NAVSIM Challenge ( 14 ) , referred to as the Navhard split, and evaluates planners using a two-stage pseudo-simulation protocol. Stage 1 evaluates the policy’s ability to drive in unseen nominal driving scenarios while stage 2 evaluates the policy’s ability to produce corrective behaviors. More details can be found in Appendix D . NAVSIM-v2 introduces the Extended PDM Score (EPDMS) as the primary metric to evaluate planners. EPDMS aggregates multiple metrics via a product of hard penalty terms and a weighted average of auxiliary terms: \displaystyle\mathrm{EPDMS}=\underbrace{\prod_{m\in\mathcal{M}_{\mathrm{pen}}}\mathrm{filter}_{m}(\text{agent},\,\text{human})}_{\text{penalty terms}}\;\cdot\; \displaystyle\underbrace{\left(\frac{\sum_{m\in\mathcal{M}_{\mathrm{avg}}}w_{m}\,\mathrm{filter}_{m}(\text{agent},\,\text{human})}{\sum_{m\in\mathcal{M}_{\mathrm{avg}}}w_{m}}\right)}_{\text{weighted average terms}}\,. The penalty terms \mathcal{M}_{\mathrm{pen}} include: (1) no at-fault collisions (NC), (2) drivable area compliance (DAC), (3) driving direction compliance (DDC), and (4) traffic light compliance (TLC). Since these penalties are combined multiplicatively, violating any one of them sets EPDMS to 0 for that scenario. To improve fairness, the function \mathrm{filter}_{m} ignores penalty violations when the human expert driver also violates the same rule in the corresponding scene. The weighted average terms consist of: (1) ego progress along the route ( w_{m}=5 ), (2) time-to-collision ( w_{m}=5 ), (3) history comfort ( w_{m}=2 ), (4) extended comfort ( w_{m}=2 ), and (5) lane keeping ( w_{m}=2 ). The final EPDMS score lies in [0,1] . In addition, we report the 4 -second Average Displacement Error (ADE@4s) on Stage 1, computed with respect to the ground-truth trajectory. Although the penalty terms are designed as hard constraints at evaluation time, directly optimizing a multiplicative hard-gated objective can lead to sparse learning signals and unstable optimization, as we show. We therefore treat penalty terms as soft behavioural constraints, targeting near-perfect satisfaction (0.99) while preserving informative gradients throughout training. 6.2 Model Configurations We initialize all experiments with the Qwen2.5-VL 3B Instruct model ( 4 ) , which we pre-train following the Poutine supervised finetuning (SFT) recipe ( 38 ) . We then perform additional SFT on the NAVSIM training set ( 7 ; 14 ) , reserving \approx 11 k scenes as a held-out split for RL finetuning (RLFT). Further pre-training details appear in Appendix D . For RLFT, we compare a set of GRPO-trained baselines. 1 1 1 Traffic light compliance (TLC) was almost always satisfied in preliminary experiments and is therefore omitted as an explicit objective, except when optimizing EPDMS. (1) GRPO with EPDMS : we directly optimize EPDMS as the reward, analogous in spirit to CaRL ( 20 ) (which optimizes a similar objective with PPO). Because EPDMS is multiplicative, any violation of NC/DAC/DDC/TLC collapses the reward to zero, effectively acting as a hard constraint. (2–5) Linear scalarization: we optimize a weighted sum of the three constraints and the weighted-average terms (WAT), \displaystyle R(\tau)=w_{1}\text{NC}+w_{2}\text{DAC}+w_{3}\text{DDC}+w_{4}\text{WAT}. This follows prior multi-objective GRPO setups ( 19 ; 25 ) . We evaluate both scalarization choices: scalarize rewards ( ScRew ) and scalarize advantages ( ScAdv ). We consider two weighting schemes: ( i) equal weights w_{i}=0.25 ( GRPO, ScRew/ScAdv, Equal Weights ), and (ii) an informed choice tuned from SFT behavior ( w_{1}{=}0.1 , w_{2}{=}0.8 , w_{3}{=}0.05 , w_{4}{=}0.05 ; GRPO, ScRew/ScAdv, Informed Weights ). (6–7) Constrained GRPO: we treat WAT as the main objective and enforce NC/DAC/DDC as constraints, setting all constraint satisfaction rate thresholds to 0.99 2 2 2 Note that to maintain the upper-bound form of the constraint, we instead enforce a \tilde{d}_{k}=0.01 threshold on the negation of the penalty terms NC/DAC/DDC. . We again compare scalarized rewards and scalarized advantages ( Constrained GRPO, ScRew vs. Constrained GRPO, ScAdv ). 6.3 Results Configuration Name NC \uparrow DAC \uparrow DDC \uparrow TLC \uparrow Ego Progress \uparrow Comfort \uparrow EPDMS \uparrow PDM-Closed (privileged) ( 13 ) 0.881 0.906 0.963 0.985 1.00 0.915 - SFT Checkpoint 0.800 0.756 0.869 0.981 0.878 0.963 0.444 GRPO with EPDMS ( 20 ) 0.749 0.759 0.829 0.969 0.979 0.945 0.381 GRPO, ScRew, Equal Weights ( 25 ) 0.751 0.696 0.787 0.968 0.986 0.954 0.347 GRPO, ScAdv, Equal Weights ( 19 ) 0.767 0.705 0.813 0.971 0.981 0.949 0.370 GRPO, ScRew, Informed Weights 0.754 0.695 0.791 0.969 0.986 0.954 0.350 GRPO, ScAdv, Informed Weights 0.778 0.731 0.831 0.972 0.944 0.962 0.385 Constrained GRPO, ScRew 0.810 0.784 0.866 0.976 0.939 0.952 0.436 Constrained GRPO, ScAdv (ours) 0.853 0.830 0.897 0.986 0.863 0.967 0.516 Table 2 : Stage 2 results on the NAVSIM-v2 Navhard split: We compare the different baseline reward function configurations and our proposed Constrained GRPO approach. ScRew stands for Scalarized rewards, while ScAdv stands for Scalarized Advantages. We see significant improvement in the EPDMS score with our constrained GRPO with scalarized advantages approach. Tables 1 and 2 report results on NAVSIM-v2 Stage 1 and Stage 2, respectively. For context, we also include the performance of the rule-based PDM planner ( 13 ) , whose trajectories are used to compute EPDMS. We now summarize the key findings. Behavior specification through constrained RL outperforms alternative strategies. Overall, our results indicate that constrained RL provides a more effective mechanism for behavior specification than alternative reward-design approaches. In particular, the two constrained GRPO variants (bottom rows of Tables 1 and 2 ) achieve the strongest overall performance, with constrained GRPO using scalarized advantages attaining the highest EPDMS. On in-distribution Stage 1 driving scenarios, both constrained variants substantially improve upon the SFT checkpoint, whereas the unconstrained baselines do not consistently yield gains. On the more challenging out-of-distribution Stage 2 split, all methods exhibit degraded performance, as expected. Nevertheless, the constrained GRPO models remain the most competitive overall, showcasing the usefulness of our behavior specification framework for limiting bad driving behaviors. Scalarized advantages outperforms scalarizing rewards. Across all three linear formulations (equal weights, informed weights, and constrained), scalarizing advantages yields consistently higher EPDMS than scalarizing rewards, for both Stage 1 and Stage 2. This trend aligns with our analysis in Section 4.2 : when objective components have mismatched scales, scalarizing rewards can implicitly bias the optimization toward high-variance terms, distorting the intended trade-offs. We further illustrate this effect in Figure 6 in Appendix D . Under constrained GRPO with scalarized rewards, the learned Lagrangian multipliers do not translate into effective constraint enforcement. This is most evident for drivable area compliance (DAC) and ego progress (EP), which is a proxy for the WAT objective: despite DAC being assigned nearly the maximum multiplier weight (and the main reward weight approaching zero), DAC does not reach the target satisfaction level, while EP continues to increase and only slowly decays later in training. In contrast, constrained GRPO with scalarized advantages responds rapidly to multiplier updates. DAC improves within the first \sim\!1000 training steps, while EP decreases as its multiplier is reduced, demonstrating that the Lagrangian weights exert their intended influence on the learning dynamics. Finally, around \sim\!2500 steps, the NC multiplier increases sharply as DAC becomes satisfactory but collision compliance degrades, showing that the optimizer adaptively reallocates emphasis across constraints. Together, these dynamics corroborate our theoretical analysis, suggesting that scalarized advantages preserve the effectiveness of multiplier-based constraint enforcement. Hard constraints lead to suboptimal learning dynamics. We also observe that directly optimizing EPDMS as a reward (i.e., GRPO with EPDMS) underperforms the soft-constraint formulations, yielding lower overall EPDMS on both Stage 1 and Stage 2. Intuitively, EPDMS couples constraints through a multiplicative structure, collapsing the learning signal whenever any penalty term is violated. This removes the ability to trade off and prioritize objectives smoothly during training, and prevents variance-aware balancing of component signals. In contrast, our soft-constraint formulation retains informative gradients even when constraints are partially violated, enabling more stable optimization and ultimately improved behavior specification. 7 Conclusion We introduced Constrained GRPO, a Lagrangian extension of GRPO that enforces user-specified constraints via indicator costs and learned multipliers. We showed that naively scalarizing rewards before GRPO’s group normalization implicitly reweights objectives through within-group variance and covariance, undermining constraint enforcement. In contrast, scalarizing advantages preserves the intended role of the multipliers and yields a coherent learning signal for constrained optimization. Experiments in a controlled gridworld and on an autonomous driving benchmark confirm that Constrained GRPO with scalarized advantages trains more stably, achieves higher task performance, and satisfies constraints more faithfully. An interesting direction for future work is to couple Constrained GRPO with curriculum-style mechanisms that adapt constraint thresholds over training, potentially improving exploration and enabling satisfaction of stricter constraints. 8