Token-Efficient RL for LLM Reasoning
Alan Lee, Harry Tong · 2025
Read Token-Efficient RL for LLM Reasoning by Alan Lee, Harry Tong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20834.txt.
Introduction
Alan Lee and Harry Tong train reasoning models on a token diet. Reinforcement learning for large language models usually scores every output token, which eats memory and compute. Their critic-free methods learn from a small, well-chosen slice of tokens instead, sized to fit LoRA fine-tuning. Two algorithms carry the idea: a stochastic group-relative method and a token-level prefix matching method for fine credit assignment.
On a 1.5 billion parameter Qwen model, accuracy on the SVAMP benchmark jumped from 46% to over 70%. Multi-digit multiplication also improved sharply. Plain full-token training under LoRA failed to beat the base model at all, a surprise the authors flag.
Catalog record
| CREATOR | Alan Lee, Harry Tong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-20834
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Token-Efficient RL for LLM Reasoning Alan Lee, Harry Tong Abstract We propose reinforcement learning (RL) strategies tailored for reasoning in large language models (LLMs) under strict memory and compute limits, with a particular focus on compatibility with LoRA fine-tuning. Building on early policy gradient methods with baseline subtraction, we design critic-free methods that operate on a small, informative subset of output tokens to reduce memory usage and stabilize training. We introduce S-GRPO, a stochastic variant of Group Relative Policy Optimization, and T-SPMO, a token-level prefix matching approach for fine-grained credit assignment. Applied to Qwen2-1.5B, our methods raise accuracy on the SVAMP benchmark from 46% to over 70% and show strong performance on multi-digit multiplication. Surprisingly, full-token GRPO under LoRA fails to improve over the base model, suggesting that selective token-level optimization may act as an implicit regularizer in low-parameter training regimes. † † affiliationnotice: Equal contribution 1 Introduction Large Language Models (LLMs) have demonstrated substantial improvements in mathematical and structured problem-solving through reinforcement learning (RL) fine-tuning on specific tasks ( Jaech et al. 2024 ) . However, standard RL approaches such as Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) and Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) can be impractical for researchers operating under limited computational budgets, for the following reasons: (1) both methods compute loss over the entire output trajectory, (2) they are typically designed for full model fine-tuning, and (3) PPO additionally requires a critic network, further increasing memory demands. In this work, we design and evaluate RL algorithms that are explicitly tailored to low compute constraints: Partial parameter updates, limited GPU capacity, and the need to mitigate overfitting in low-capacity adaptation settings. First, we introduce Stochastic-GRPO (S-GRPO), a lightweight variant of GRPO that samples tokens from output trajectories to contribute to the loss. Second, we propose Token-Specific Prefix Matching Optimization (T-SPMO), which assigns credit at a token granularity. Our methods extend early baseline-based policy gradients ( Kool et al. 2019 ; Williams 1992 , e.g.,) to a fine-grained token-level setting under LoRA constraints. To evaluate these methods, we fine-tune the open-weight Qwen2-1.5B model ( Yang et al. 2024 ) using LoRA ( Hu et al. 2021 ) on two reasoning benchmarks: the SVAMP dataset ( Patel et al. 2021 ) and a multi-digit multiplication task. Despite updating the policy using only 30–50% of generated tokens in S-GRPO and under 5% in T-SPMO, both methods substantially improve test-set accuracy across both benchmarks. We also establish a full-token GRPO baseline under LoRA, which surprisingly did not improve performance from the base model. These results suggest that carefully designed, memory-efficient RL algorithms can significantly improve LLM reasoning under compute constraints—even outperforming exhaustive full-token optimization strategies. 2 Related Work 2.1 RL for Language Models Reinforcement learning has been widely used to align language model outputs with task-specific or user-defined objectives. Many RL algorithms used to fine-tune LLMs, such as PPO ( Schulman et al. 2017 ) and ReST-MCTS ( Zhang et al. 2024 ) , require training secondary models, such as critic or process reward models, increasing compute demands. Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) removes the critic by using group-level reward normalization, yet still relies on full-model fine-tuning and full-sequence loss computation, limiting its applicability under compute constraints. 2.2 Baselines and Grouped Advantage Estimation Classic policy gradient methods reduce variance by subtracting a baseline from the return ( Williams 1992 ) . Later work ( Kool et al. 2019 ) demonstrated that using average rewards over sampled completions—rather than a learned value function—can yield strong performance. We extend this idea to token-level credit assignment for LLMs. 2.3 Memory-Efficient Fine-Tuning Techniques like Low-Rank Adaptation (LoRA) ( Hu et al. 2021 ) enable parameter-efficient adaptation of large models on modest hardware by training low-rank adapters in frozen networks. We focus on RL methods that are compatible with LoRA. 3 Methods 3.1 Problem Setup We focus on enhancing the reasoning capabilities of large language models (LLMs) using RL under constrained computational resources. The task is framed as conditional generation with reward feedback , where the model generates a sequence of tokens Y=(y_{1},y_{2},\dots,y_{T}) in response to a prompt x , and receives a scalar reward signal r(Y)\in\mathbb{R} based on the correctness or quality of the generated sequence. Let \pi_{\theta}(y_{t}\mid x,y_{<t}) denote the autoregressive policy defined by the language model with parameters \theta . The objective is to optimize \theta to maximize the expected reward over the distribution of completions: \mathcal{J}(\theta)=\mathbb{E}_{Y\sim\pi_{\theta}}\left[r(Y)\right]. (1) Traditional policy gradient methods often rely on estimating the advantage function using a separate value network, which can be computationally intensive and memory-demanding. To address these challenges, the recently popularized critic-free RL algorithm Group Relative Policy Optimization (GRPO) estimates advantages by comparing the rewards of multiple responses generated for the same prompt ( Shao et al. 2024 ) . For each input prompt, GRPO generates a group \mathcal{G} completions from the current policy \pi_{\theta} . Each completion y^{(i)} in group i is tokenized as y^{(i)}=(y^{(i)}_{1},y^{(i)}_{2},\dots,y^{(i)}_{T_{i}}) and assigned a scalar reward R_{i} . The advantage of each token is normalized using the group-level statistics: \hat{A}_{i,t}=\frac{R_{i}-\mu_{R}}{\sigma_{R}} (2) Where \mu_{R} and \sigma_{R} are the mean and standard deviation of rewards across the group. The policy is updated to increase the likelihood of responses with positive advantages and decrease it for those with negative advantages: \displaystyle\mathcal{J}(\theta) \displaystyle=\frac{1}{|\mathcal{G}|}\sum_{i=1}^{|\mathcal{G}|}\frac{1}{|y^{(i)}|}\sum_{t=1}^{|y^{(i)}|}\Biggl\{\min\Biggl[\frac{\pi_{\theta}\!\bigl(y_{t}^{(i)}\!\mid\!x,y_{<t}^{(i)}\bigr)}{\pi_{\theta_{\text{old}}}\!\bigl(y_{t}^{(i)}\!\mid\!x,y_{<t}^{(i)}\bigr)}\,\hat{A}_{i,t}, (3) \displaystyle\text{clip}\!\Bigl(\frac{\pi_{\theta}\!\bigl(y_{t}^{(i)}\!\mid\!x,y_{<t}^{(i)}\bigr)}{\pi_{\theta_{\text{old}}}\!\bigl(y_{t}^{(i)}\!\mid\!x,y_{<t}^{(i)}\bigr)},1-\epsilon,\,1+\epsilon\Bigr)\hat{A}_{i,t}\Biggr] \displaystyle-\,\beta\,\mathbb{D}_{\text{KL}}\!\bigl[\pi_{\theta}\,\|\,\pi_{\text{ref}}\bigr]\Biggr\} Where \pi_{\text{ref}} and \pi_{\text{old}} are previous versions of \pi_{\theta} , which are updated to \pi_{\theta} after varying iterations. \epsilon is a hyperparameter for PPO-style objective clipping, and \beta is a hyperparameter coefficient for the KL-divergence regularization. GRPO estimates the KL-Divergence using the unbiased estimator: \displaystyle\mathrm{D}_{\mathrm{KL}}\left[\pi_{\theta}\,\|\,\pi_{\text{ref}}\right] \displaystyle=\frac{\pi_{\text{ref}}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})}{\pi_{\theta}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})} \displaystyle\quad-\log\frac{\pi_{\text{ref}}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})}{\pi_{\theta}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})}-1 (4) To further accommodate limited computational resources, we employ Low-Rank Adaptation (LoRA) ( Hu et al. 2021 ) for parameter-efficient fine-tuning. LoRA allows us to fine-tune only a small subset of the model’s parameters, significantly reducing memory requirements. We evaluate our methods on two representative reasoning tasks: • SVAMP , a benchmark of verbal arithmetic problems requiring symbolic and numerical reasoning ( Patel et al. 2021 ) . • Multi-digit multiplication , where models must learn step-wise calculation strategies. 3.2 S-GRPO: Stochastic Group Relative Policy Optimization Stochastic Group Relative Policy Optimization (S-GRPO) extends GRPO to a low-memory setting by reducing the tokens that contribute to the gradient from the full response trajectory. We also set the number of updates per problem (denoted as \mu in the original GRPO paper ( Shao et al. 2024 ) ) to 1, aligning with the default setting in lightweight RL libraries such as TRL. This choice both simplifies optimization by eliminating the need for PPO-style clipped objectives and reduces computational overhead, consistent with our focus on efficient fine-tuning. Also in alignment of the default setting in TRL, we do not update \pi_{\text{ref}} to \pi_{\theta} . The policy objective is thus: \displaystyle\mathcal{J}_{\text{SGRPO}}(\theta) \displaystyle=\frac{1}{|\mathcal{G}|}\sum_{i=1}^{\mathcal{G}}\frac{1}{|\mathcal{T}_{i}|}\sum_{t\in\mathcal{T}_{i}}(\frac{\pi_{\theta}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})}{\pi_{\theta}(y^{(i)}_{t}\mid x,y^{(i)}_{<t})\big|_{\text{no grad}}}\hat{A}_{i,t} \displaystyle\quad-\,\beta\,D_{\text{KL}}\!\left[\pi_{\theta}\;\|\;\pi_{\text{ref}}\right]) (5) Here, \mathcal{T}_{i} denotes the set of tokens in completion i that contribute to the loss. Tokens are included in \mathcal{T}_{i} according to: \displaystyle t\in\mathcal{T}_{i}\iff\begin{cases}1&\text{if }t<\alpha\\ \text{Bernoulli}(P)&\text{if }t\geq\alpha\text{ and }|\mathcal{T}_{i}|<k\\ 0&\text{otherwise}\end{cases} (6) where \alpha is a cutoff index ensuring early tokens are always included, k is the maximum number of tokens contributing