FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning

Yizhou Zhang, Ning Lv, Teng Wang, Jisheng Dang · 2025

Read FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning by Yizhou Zhang, Ning Lv, Teng Wang, Jisheng Dang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21792.txt.

Introduction

Yizhou Zhang, Ning Lv, Teng Wang, and Jisheng Dang speed up a popular training method for language models in this 2025 paper. The method, called GRPO, improves reasoning by scoring groups of answers against each other. Training is slow because the model must write many answers for each question. The authors target that writing phase, which is the main bottleneck.

Their system, FastGRPO, guesses ahead with a small draft model, a trick called speculative decoding. It adjusts the guessing plan to match how busy the training run is at each moment. A second trick lets the draft model keep learning from the main model's feedback, so the two do not drift apart. On math reasoning tests across several models, training ran well over twice as fast from end to end, beating simpler approaches.

Catalog record

CREATORYizhou Zhang, Ning Lv, Teng Wang, Jisheng Dang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-21792

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learning Yizhou Zhang, Ning Lv, Teng Wang, Jisheng Dang Abstract Group relative policy optimization (GRPO) has demonstrated significant potential in improving the reasoning capabilities of large language models (LLMs) via reinforcement learning. However, its practical deployment is impeded by an excessively slow training process, primarily attributed to the computationally intensive autoregressive generation of multiple responses per query, which makes the generation phase the primary performance bottleneck. Although speculative decoding presents a promising direction for acceleration, its direct application in GRPO achieves limited speedup under high-concurrency training conditions. To overcome this limitation, we propose a concurrency-aware speculative decoding framework that dynamically adjusts the drafting and verification strategy according to real-time concurrency levels, thereby maximizing the acceleration of the generation process. Furthermore, to address performance degradation arising from distributional drift between the evolving target model and the fixed draft model during training, we introduce an online draft learning mechanism that enables the draft model to continuously adapt using feedback signals from the target model. Experimental results across multiple mathematical reasoning datasets and models demonstrate that the proposed method achieves end-to-end speedups of 2.35x to 2.72x, significantly surpassing baseline approaches in efficiency. The code is available at https://github.com/yedaotian9/GRPO_speculative . 1 Introduction Group relative policy optimization (GRPO) has recently emerged as a promising framework for enhancing the reasoning capabilities of large language models (LLMs) through reinforcement learning Team 2025a . In each training iteration, the LLM generates a group of responses to a given query. These responses are subsequently evaluated using a predefined rule-based reward function, and the resulting rewards are standardized prior to model updates via policy optimization Shao et al. 2024 . This approach exploits group-level feedback signals to guide the model toward more accurate and coherent reasoning behaviors. However, compared to conventional supervised fine-tuning (SFT) Ouyang et al. 2022 , the GRPO training paradigm suffers from significantly lower throughput, hindering its adoption and limiting empirical exploration. This bottleneck arises primarily from the autoregressive inference required for response generation, which dominates the training pipeline. As shown in Figure 1 (a), the generation phase (i.e., response sampling) accounts for 91% to 98% of total training time across multiple mathematical reasoning datasets, making it the primary target for performance optimization. To this end, we propose integrating speculative decoding , a compute-efficient and accuracy-preserving technique originally introduced by Leviathan et al. 2023 for accelerating autoregressive inference, into the GRPO framework. Speculative decoding employs a smaller, faster ”draft model” to generate candidate token sequences in advance, which are then efficiently verified by the larger target model. This method capitalizes on the observation that autoregressive decoding is typically memory-bound rather than compute-bound, thereby enabling increased computational parallelism and improved hardware utilization. By reducing the number of direct forward passes through the target model, speculative decoding offers the potential for substantial acceleration of the generation phase in GRPO. However, direct integration of speculative decoding into the GRPO generation phase yields significantly lower speedup compared to results reported in prior work Weng et al. 2025 ; Li et al. 2025a , and in certain cases even leads to performance degradation. This discrepancy arises because speculative decoding is predominantly evaluated in low-concurrency, low-latency settings (e.g., edge device deployment), whereas the generation phase in GRPO operates under high-concurrency and high-latency conditions. Although speculative decoding alleviates memory bandwidth pressure, it introduces additional computational overhead. In high-concurrency scenarios, this trade-off may shift the system from a memory-bound to a compute-bound regime, thereby negating the expected performance gains. To address this challenge, we analyze the responses generated during each step of the GRPO training process and observe significant variability in sequence lengths across batches. This variability leads to a substantial reduction in effective concurrency during the generation phase, with effective concurrency declining from a high initial batch size to nearly one as sequences terminate at different times due to uneven completion. Motivated by this observation, we propose a concurrency-aware speculative decoding strategy specifically designed for the generation phase in GRPO. Our method dynamically adjusts its configuration (i.e., the draft tree size and the number of verification tokens) based on real-time concurrency levels. Consequently, it achieves moderate speedup during the early, high-concurrency stage and delivers progressively greater acceleration as concurrency decreases, thereby minimizing the overall latency of the generation phase. Furthermore, we identify a critical challenge inherent in the GRPO training framework: the target model undergoes continuous parameter updates, resulting in an increasing divergence between the target model and the fixed draft model. This distributional shift inevitably degrades model alignment, leading to declining token acceptance rates and diminishing acceleration gains over time. To mitigate this issue, we propose an online draft learning strategy within the GRPO loop. Our approach updates the draft model using online feedback signals derived from the evolving target model. This continual adaptation significantly enhances the draft model’s representational fidelity, thereby increasing the average length of accepted speculative tokens and sustaining higher acceleration ratios throughout training, rather than allowing them to deteriorate. We conduct experiments on Qwen2.5-7B-Instruct, Llama3.1-8B-Instruct, and other models across three mathematical reasoning datasets of varying difficulty: GSM8K Cobbe et al. 2021 , SimpleRL-Abel-Level3to5 Zeng et al. 2025 , and DAPO-Math-17K Yu et al. 2025 . As illustrated in Figure 1 (b), our proposed method substantially improves inference acceleration, achieving end-to-end speedups of 2.35x to 2.72x across different models and datasets. Figure 1: (a) Proportion of generation time during GRPO training across multiple models and datasets, showing that generation dominates the overall time cost. (b) Speedup ratios on various models and datasets, demonstrating the significant and broad acceleration achieved by our approach. 2 Preliminaries Speculative Decoding. Speculative decoding Leviathan et al. 2023 is an inference acceleration technique for autoregressive language models that leverages a fast draft model to propose candidate token sequences, which are then verified in parallel by a more capable but slower target model . The method is motivated by two key observations: (1) not all generation steps require the full capacity of the target model; some tokens can be predicted accurately by smaller models Hinton et al. 2015 ; Jaszczur et al. 2021 ; Hubara et al. 2016 ; So et al. 2021 ; Shazeer 2019 ; and (2) in modern hardware, inference is often limited by memory bandwidth rather than compute Vaswani et al. 2017 ; Park et al. 2025 , leaving room for additional parallel computation. The process operates in two phases. First, the draft model autoregressively generates several candidate tokens. Then, the target model performs a single forward pass over the input prefix concatenated with these speculative tokens, using an appropriate attention mask to compute logits for all positions in parallel. Drafted tokens are only accepted if they match the target model’s output. Draft Model Architectures. We adopt the draft model architecture from EAGLE Li et al. 2024b enhanced by Feature Sampling and Partial Alignment Distillation Gui et al. 2024 . This approach trains a standalone draft model to autoregressively generate hidden states, which are then decoded into tokens using the LM head of the target LLM. The draft model takes the previously sampled tokens as input to stabilize generation, enabling effective feature-level modeling while leveraging the target model’s output distribution. Verification Strategies.

Read the complete text