Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, Xing Yu · 2026

Read Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning by Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, Xing Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10535.txt.

Introduction

Zichao Li and co-authors tackle length inflation in reinforcement learning for large language models, where models pad reasoning to chase rewards. Past fixes add penalties that create shortcuts, or gate heuristically in ways that do not generalize. Their method rescales rewards multiplicatively in a group-relative way, forming a continuous gate that adapts to reward level. Added regularization tunes length budgets to task difficulty and keeps strong trajectories' advantage intact. Across RLHF and RLVR settings it matched standard GRPO performance while cutting the bloat, beating leading baselines.

Catalog record

CREATORZichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, Xing Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70072
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10535

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning Zichao Li, Jie Lou, Fangchen Dong, Zhiyuan Fan, Mengjie Ren, Hongyu Lin, Xianpei Han, Debing Zhang, Le Sun, Yaojie Lu, Xing Yu Abstract Reinforcement learning significantly enhances LLM capabilities but suffers from a critical issue: length inflation , where models adopt verbosity or inefficient reasoning to maximize rewards. Prior approaches struggle to address this challenge in a general and lossless manner, primarily because additive penalties introduce a compensatory effect that creates optimization shortcuts, while heuristic gating strategies lack generality beyond binary feedback. To bridge this gap, we present Group Relative Reward Rescaling ( GR 3 ) , which reframes length control as a multiplicative rescaling paradigm, effectively establishing a generalized, continuous, and reward-dependent gating mechanism. To further ensure lossless optimization, we incorporate group-relative regularization and advantage-aware calibration, which dynamically adapt length budgets to instance difficulty and preserve the advantage signal of high-quality trajectories. Empirically, across both RLHF and RLVR settings, GR 3 maintains training dynamics and downstream performance comparable to standard GRPO while significantly mitigating length inflation, outperforming state-of-the-art length-regularized baselines. 1 Introduction Figure 1: Comparison of GR 3 with open-source efficient reasoning models, all trained on DeepSeek-R1-Distill-7B. GR 3 pioneers a new paradigm that sustains stable performance gains under RL while simultaneously mitigating the length inflation issue. Reinforcement learning (RL) ( 5 ; 15 ) has become the engine of post-training for Large Language Models (LLMs) ( 1 ; 41 ) . Yet this engine exhibits a persistent flaw, which we term length inflation : a tendency for RL-trained models to produce unnecessarily lengthy trajectories, inflating inference costs without proportional gains in quality. This phenomenon arises across major RL paradigms. In RL from human feedback (RLHF) ( 33 ) , models exploit reward-model biases toward verbosity, leading to reward hacking ( 14 ) . In RL with verifiable rewards (RLVR) ( 35 ) , length inflation instead stems from reasoning inefficiency ( 40 ) , where models generate unnecessarily long chains of thought to marginally improve the likelihood of a correct solution. Figure 2: Training dynamics of GR 3 , which retains GRPO’s reward gains without loss while significantly reducing average tokens. The base models used for the two settings are DeepSeek-R1-Distill-1.5B and Qwen3-8B (without thinking mode), respectively. Prior work has sought to mitigate length inflation. One line of research trains reward models that are invariant to response length ( 7 ; 24 ) . While effective in RLHF, this strategy does not extend to RLVR, where rewards are derived from ground-truth verifiers rather than learned proxies that can be debiased. A more general direction instead introduces explicit length penalties into the reward ( 26 ; 25 ; 47 ) . However, most existing methods rely on coarse regularization, which leads to suboptimal optimization dynamics. A common design adopts additive shaping ( 48 ; 42 ) , modifying the objective with an explicit length term (e.g., R^{\prime}=R-\lambda\ell ). This introduces decoupled incentives, creating a length-driven component that makes extreme brevity an attractive shortcut independent of task success. To better align penalties with outcomes, some works propose heuristic gating ( 9 ; 4 ) , applying penalties only when R=1 . However, such designs are inherently limited to binary feedback and do not extend naturally to continuous-reward settings like RLHF. Moreover, many approaches rely on coarse control mechanisms, such as static truncation thresholds or uncalibrated penalty strengths ( 23 ; 9 ) , resulting in an inherent efficiency–performance trade-off, as illustrated in Figure 1 . These observations lead to a central question: Can we tackle length inflation in a general manner without compromising the capability gains of RL? In this work, we present Group Relative Reward Rescaling ( GR 3 ) , a principled framework for lossless efficiency optimization. Rather than using additive penalties, GR 3 regularizes length through multiplicative rescaling , which acts as a generalized gating mechanism and removes the compensatory shortcuts inherent to additive schemes. To further ensure lossless optimization, we introduce two fine-grained mechanisms. Specifically, we employ group-relative regularization , which normalizes length against on-policy statistics rather than rigid thresholds, thereby dynamically adapting the length budget to the inherent difficulty of each prompt. Complementing this, we introduce advantage-aware calibration to explicitly control the penalty strength. This ensures that length regularization does not overturn the advantage signal of representative high-quality trajectories, thereby safeguarding stable optimization toward capability improvements. Empirically, GR 3 resolves the efficiency–performance trade-off inherent in prior methods. As shown in Figure 1 , our approach significantly reduces token usage (e.g., over 40\% on AIME-25) while simultaneously improving accuracy (e.g., +8 points), demonstrating that verbosity is not a prerequisite for intelligence. Furthermore, in RLHF settings, GR 3 exhibits an adaptive length dynamic: it permits moderate growth when computation is beneficial but automatically curtails generation length as the policy matures (Figure 2 ). This mechanism effectively mitigates reward hacking via verbosity without sacrificing capability. We will release our code and model checkpoints to support future research. In summary, our contributions are threefold: • We propose GR 3 , a framework for lossless length control that substitutes additive penalties with multiplicative reward rescaling. This design eliminates compensatory optimization shortcuts and provides a unified mechanism for both binary and continuous rewards. • We develop an optimization-preserving strategy that integrates group-relative regularization with advantage-aware calibration, adapting constraints to on-policy statistics while preserving learning signals. • Across mathematical reasoning, code generation, and RLHF alignment tasks, GR 3 yields concise generations while matching standard GRPO performance, shifting the efficiency–performance Pareto frontier. 2 Preliminary 2.1 Group Relative Policy Optimization LLM generation can be formulated as a token-level Markov Decision Process ( 34 ) . Given a prompt x\sim\mathcal{D} , an autoregressive policy \pi_{\theta} generates a response y=(y_{1},\dots,y_{\ell}) of length \ell:=|y| by sampling tokens from \pi_{\theta}(y_{t}\mid x,y_{<t}) . A scalar reward R(x,y) is defined over complete responses, and reinforcement learning aims to maximize the expected reward: \max_{\pi_{\theta}}\;\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_{\theta}(\cdot\mid x)}\big[R(x,y)\big]. (1) With the emergence of reasoning models such as DeepSeek-R1 ( 15 ) , group-style RL has become prevalent for LLM post-training. Among them, Group Relative Policy Optimization (GRPO) ( 35 ) is widely adopted due to its scalability and its elimination of the need for a separate value model. For each prompt x , GRPO samples a group of G responses \{y^{(i)}\}_{i=1}^{G} from an old policy \pi_{\theta_{\mathrm{old}}}(\cdot\mid x) and evaluates each by R(x,y^{(i)}) . It then constructs a group-relative advantage via within-group normalization: \begin{gathered}\hat{A}^{(i)}=\frac{R(x,y^{(i)})-\mu_{R}}{\sigma_{R}}\\[4.0pt] \begin{aligned} \mu_{R}&:=\frac{1}{G}\sum_{j=1}^{G}R(x,y^{(j)}),\sigma_{R}:=\mathrm{std}\!\left(\{R(x,y^{(j)})\}_{j=1}^{G}\right)\end{aligned}\end{gathered} (2) The policy is optimized using a PPO-style clipped objective over the group-normalized advantages: \displaystyle\mathcal{J}_{\mathrm{GRPO}}(\theta) \displaystyle=\mathbb{E}_{x\sim\mathcal{D},\,\{y^{(i)}\}_{i=1}^{G}}\Bigg[\frac{1}{G}\sum_{i=1}^{G}\sum_{t=1}^{|y^{(i)}|} (3) \displaystyle\Bigl(\min\big(r_{i,t}(\theta)\hat{A}^{(i)},\,\mathrm{clip}(r_{i,t}(\theta),1-\varepsilon,1+\varepsilon)\hat{A}^{(i)}\big) \displaystyle-\beta D_{\mathrm{KL}}(\pi_{\theta}\|\pi_{\mathrm{ref}})\Bigr)\Bigg], where the importance sampling ratio is defined as r_{i,t}(\theta)=\frac{\pi_{\theta}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})}{\pi_{\theta_{\mathrm{old}}}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})}. (4) Notably, GRPO estimates advantages based on group-wise statistics. We exploit this structural property to construct a more on-policy length regularization scheme and to formulate advantage-aware calibration that better respect the underlying optimization signal, as detailed in Section 3 . Table 1: Comparison of length-aware reward shaping methods ( 25 ; 4 ; 2 ; 42 ; 48 ; 9 ) . \ell^{(i)} is the response length of sample i . \ell_{\min} , \ell_{\max} , \bar{\ell} , and \sigma_{\ell} are computed within each group. \alpha , \lambda , \ell_{T} and \ell_{C} are fixed hyperparameters. s(\cdot) denotes

Read the complete text