StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation

Yinmin Zhong 1 {}^{\text{1}} Zili Zhang 1 {}^{\text{1}} Xiaoniu Song 2 {}^{\text{2}} Hanpeng Hu 2 {}^{\text{2}} Chao Jin 1 {}^{\text{1}} Bingyang Wu 1 {}^{\text{1}} Nuo Chen 2 {}^{\text{2}} Yukun Chen 2 {}^{\text{2}} Yu Zhou 2 {}^{\text{2}} Changyi Wan 2 {}^{\text{2}} Hongyu Zhou 2 {}^{\text{2}} Yim · 2025

Read StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation by Yinmin Zhong 1 {}^{\text{1}} Zili Zhang 1 {}^{\text{1}} Xiaoniu Song 2 {}^{\text{2}} Hanpeng Hu 2 {}^{\text{2}} Chao Jin 1 {}^{\text{1}} Bingyang Wu 1 {}^{\text{1}} Nuo Chen 2 {}^{\text{2}} Yukun Chen 2 {}^{\text{2}} Yu Zhou 2 {}^{\text{2}} Changyi Wan 2 {}^{\text{2}} Hongyu Zhou 2 {}^{\text{2}} Yim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15930.txt.

Introduction

Yinmin Zhong and thirteen co-authors rebuild the factory floor for teaching language models with rewards, arguing the two halves of the job should not share the same machines, in this 2025 paper. Reinforcement learning for chatbots alternates: generate sample answers, then train on the scored results. Conventional systems colocate both stages on the same hardware, taking turns.

That sharing couples their appetites badly at scale. Generation wants memory and bursts; training wants steady compute. StreamRL separates them, disaggregated by design, and attacks the two diseases of separated pipelines. Pipeline bubbles, idle gaps while one stage waits for the other, are dissolved by streaming generation, letting training feed continuously. Skewness bubbles, stalls from a few very long answers, are handled by the elastic scheduling the split makes possible. The result claims the flexibility of separate clusters and heterogeneous hardware, without the dead air that used to punish the design.

Catalog record

CREATORYinmin Zhong 1 {}^{\text{1}} Zili Zhang 1 {}^{\text{1}} Xiaoniu Song 2 {}^{\text{2}} Hanpeng Hu 2 {}^{\text{2}} Chao Jin 1 {}^{\text{1}} Bingyang Wu 1 {}^{\text{1}} Nuo Chen 2 {}^{\text{2}} Yukun Chen 2 {}^{\text{2}} Yu Zhou 2 {}^{\text{2}} Changyi Wan 2 {}^{\text{2}} Hongyu Zhou 2 {}^{\text{2}} Yim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15930

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation Yinmin Zhong 1 {}^{\text{1}} Zili Zhang 1 {}^{\text{1}} Xiaoniu Song 2 {}^{\text{2}} Hanpeng Hu 2 {}^{\text{2}} Chao Jin 1 {}^{\text{1}} Bingyang Wu 1 {}^{\text{1}} Nuo Chen 2 {}^{\text{2}} Yukun Chen 2 {}^{\text{2}} Yu Zhou 2 {}^{\text{2}} Changyi Wan 2 {}^{\text{2}} Hongyu Zhou 2 {}^{\text{2}} Yimin Jiang 3 {}^{\text{3}} Yibo Zhu 2 {}^{\text{2}} Daxin Jiang 2 {}^{\text{2}} 1 {}^{\text{1}} School of Computer Science, Peking University 2 {}^{\text{2}} StepFun 3 {}^{\text{3}} Unaffiliated Abstract . Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs). RL for LLMs involves two stages: generation and training. The LLM first generates samples online, which are then used to derive rewards for training. The conventional view holds that the colocated architecture —where the two stages share resources via temporal multiplexing—outperforms the disaggregated architecture , in which dedicated resources are assigned to each stage. However, in real-world deployments, we observe that the colocated architecture suffers from resource coupling , where the two stages are constrained to use the same resources. This coupling compromises the scalability and cost-efficiency of colocated RL in large scale training. In contrast, the disaggregated architecture allows for flexible resource allocation, supports heterogeneous training setups, and facilitates cross-datacenter deployment. StreamRL is designed with disaggregation from first principles and fully unlocks its potential by addressing two types of performance bottlenecks in existing disaggregated RL frameworks: pipeline bubbles , caused by stage dependencies, and skewness bubbles , resulting from long-tail output length distributions. To address pipeline bubbles, StreamRL breaks the traditional stage boundary in synchronous RL algorithms through stream generation, and achieves fully overlapping in asynchronous RL. To address skewness bubbles, StreamRL employs an output-length ranker model to identify long-tail samples and reduces generation time via skewness-aware dispatching and scheduling. Experiments show that StreamRL improves throughput by up to 2.66 × 2.66\times compared to existing state-of-the-art systems, and improves cost-effectiveness by up to 1.33 × 1.33\times in heterogeneous, cross-datacenter setting. 1. Introduction Reinforcement learning (RL) has emerged as a new paradigm for training large language models (LLMs), substantially improving their reasoning capabilities and revealing a novel inference-time scaling law ( DeepSeek-AI et al., 2025 ; Yu et al., 2025 ; see, 2025 ) . State-of-the-art models such as OpenAI o1 ( ope, 2024a ) and o3 ( ope, 2024b ) , Claude 3.7 Sonnet ( cla, 2025 ) , and DeepSeek-R1 ( DeepSeek-AI et al., 2025 ) have all adopted RL to achieve leading performance in tasks such as coding and mathematics. In contrast to traditional next-token prediction ( Vaswani et al., 2017 ; Brown et al., 2020 ; OpenAI, 2023 ; Chowdhery et al., 2022 ; Touvron et al., 2023 ) in pre-training, RL enables the LLMs to learn by trial and error from reward signals. While numerous RL algorithms exist, such as PPO ( Schulman et al., 2017 ) and GRPO ( Shao et al., 2024 ) , the typical RL workflow for LLMs involves two main stages in serial: generation and training. In generation stage, the LLM produces samples on a batch of given prompts, followed by the training stage that updates the LLM based on rewards derived from the generated samples. Due to this two-stage workflow, initial RL training frameworks for LLMs, such as OpenRLHF ( Hu et al., 2024 ) and NeMo ( Harper et al., 2025 ) , naturally adopted a disaggregated architecture . As depicted in Figure 1 (a), dedicated computational resources are allocated for each stage separately. The generation stage employs existing inference frameworks like vLLM ( Kwon et al., 2023a ) to generate samples, which are subsequently transferred to the training stage that uses training frameworks like DeepSpeed ( Aminabadi et al., 2022 ) or Megatron-LM ( Shoeybi et al., 2020 ) . Updated model weights are then transferred back to the inference framework for the next iteration generation. This architectural choice can effectively reuse existing infrastructures and facilitate rapid deployment of various RL algorithms, which gains broad initial adoption. However, a notable drawback of disaggregated architecture is resource idleness arising from serial dependency: GPU resources allocated to the training stage remain idle during the generation stage, and vice versa. Figure 1 . Two representative RL framework architectures. To address this inefficiency, recent RL training frameworks, such as verl ( Sheng et al., 2024a ) , ReaL ( Mei et al., 2024 ) , and RLHFuse ( Zhong et al., 2024b ) , have adopted a colocated architecture . As illustrated in Figure 1 (b), it colocates the generation and training stages on the same GPU resources. When one stage is active, the states of the other stage (such as model weights and optimizer states) is temporarily stored in CPU memory. Context switching happens between stage boundary, enabling time-division multiplexing of GPU resources. This architectural shift resolves the resource idleness issue and improves training efficiency. Subsequently, colocation becames the prevailing choice and was widely acknowledged to be superior to disaggregation. We initially believed the same and chose colocated architecture for our internal RL framework. However, in practical deployment, we have observed that the colocated architecture encounters the problem of resource coupling as the training scales out. The reason lies in that the two stages feature fundamentally distinct workloads: the generation stage is notably memory-bandwidth-bound ( Zhong et al., 2024a ) , whereas the training stage is typically compute-bound ( Jiang et al., 2024 ) . However, due to colocation, both stages must share identical resource quantities and hardware types , creating an inherent conflict with their divergent computational characteristics. Concretely, the performance speedup for the generation stage reaches a plateau much more quickly compared to the compute-bound training stage when scaling out resources. Yet, the colocated architecture restricts resource quantities for both stages to be identical, thus diminishing overall resource utilization. Also, it is unable to select the most suitable and cost-effective hardware types for each stage respectively. Moreover, constrained by factors such as policy, cost, and power supply, constructing a single large-scale datacenter can be challenging and expensive ( mul, 2024a ) . Consequently, companies typically operate multiple medium-sized datacenters equipped with GPUs spanning different generations and types, forming a cross-datacenter heterogeneous resource pool ( mul, 2024b ) . As RL training scales out, the colocated architecture struggles to efficiently leverage this entire resource pool, as the training stage typically involves full-mesh communication operations ( Shou et al., 2025 ) , which will incur significant communication overhead across datacenters. Under such circumstances, the initial disaggregated architecture reshines. First, resource quantities for the generation and training stages need not be identical, allowing more flexible and efficient resource allocation. Second, it allows for selecting the most suitable hardware for each stage, such as adopting more cost-effective inference GPUs for the generation stage, while using more expensive and high-performance GPUs exclusively for the training stage. Moreover, RL features point-to-point data transfer between the two stages, keeping the communication overhead manageable even across datacenters. By putting the two stages in separate datacenters, RL training can overcome the constraints of a single homogeneous datacenter and fully leverage the entire cross-datacenter, heterogeneous resource pool. Nevertheless, the disaggregated architecture encounters two challenges in existing frameworks, preventing it from

Read the complete text