RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs
Yongji Wu 1 , ∗ {}^{\text{1},*} Xueshen Liu 2 , 3 , ∗ , † {}^{\text{2},\text{3},*,\text{\textdagger}} Haizhong Zheng 4 {}^{\text{4}} Juncheng Gu 3 {}^{\text{3}} Beidi Chen 4 {}^{\text{4}} Z. Morley Mao 3 {}^{\text{3}} Arvind Krishnamurthy 3 , 5 {}^{\text{3},\text{5}} Ion Stoica 1 {}^{\text{1}} 1 {}^ · 2025
Read RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs by Yongji Wu 1 , ∗ {}^{\text{1},*} Xueshen Liu 2 , 3 , ∗ , † {}^{\text{2},\text{3},*,\text{\textdagger}} Haizhong Zheng 4 {}^{\text{4}} Juncheng Gu 3 {}^{\text{3}} Beidi Chen 4 {}^{\text{4}} Z. Morley Mao 3 {}^{\text{3}} Arvind Krishnamurthy 3 , 5 {}^{\text{3},\text{5}} Ion Stoica 1 {}^{\text{1}} 1 {}^ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19225.txt.
Introduction
Yongji Wu and seven co-authors present RLBoost, a system that harvests spare, preemptible computing capacity to cut the cost of reinforcement learning for large language models. Such training alternates between generating responses, which scales easily across many machines, and updating the model, which needs tightly linked graphics processors.
Older designs either force both stages onto the same hardware or split them and leave resources idle. RLBoost instead feeds the generation stage with cheap capacity that can be reclaimed at any time, keeping the expensive training hardware busy. Cheaper post-training matters as reinforcement learning becomes a standard step for reasoning models.
Catalog record
| CREATOR | Yongji Wu 1 , ∗ {}^{\text{1},*} Xueshen Liu 2 , 3 , ∗ , † {}^{\text{2},\text{3},*,\text{\textdagger}} Haizhong Zheng 4 {}^{\text{4}} Juncheng Gu 3 {}^{\text{3}} Beidi Chen 4 {}^{\text{4}} Z. Morley Mao 3 {}^{\text{3}} Arvind Krishnamurthy 3 , 5 {}^{\text{3},\text{5}} Ion Stoica 1 {}^{\text{1}} 1 {}^ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19225
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs Yongji Wu 1 , ∗ {}^{\text{1},*} Xueshen Liu 2 , 3 , ∗ , † {}^{\text{2},\text{3},*,\text{\textdagger}} Haizhong Zheng 4 {}^{\text{4}} Juncheng Gu 3 {}^{\text{3}} Beidi Chen 4 {}^{\text{4}} Z. Morley Mao 3 {}^{\text{3}} Arvind Krishnamurthy 3 , 5 {}^{\text{3},\text{5}} Ion Stoica 1 {}^{\text{1}} 1 {}^{\text{1}} UC Berkeley 2 {}^{\text{2}} University of Michigan 3 {}^{\text{3}} Google 4 {}^{\text{4}} CMU 5 {}^{\text{5}} University of Washington Abstract . Reinforcement learning (RL) has become essential for unlocking advanced reasoning capabilities in large language models (LLMs). RL workflows involve interleaving rollout and training stages with fundamentally different resource requirements. Rollout typically dominates overall execution time, yet scales efficiently through multiple independent instances. In contrast, training requires tightly-coupled GPUs with full-mesh communication. Existing RL frameworks fall into two categories: co-located and disaggregated architectures. Co-located ones fail to address this resource tension by forcing both stages to share the same GPUs. Disaggregated architectures, without modifications of well-established RL algorithms, suffer from resource under-utilization. Meanwhile, preemptible GPU resources, i.e., spot instances on public clouds and spare capacity in production clusters, present significant cost-saving opportunities for accelerating RL workflows, if efficiently harvested for rollout. In this paper, we present RLBoost ‡ , a systematic solution for cost-efficient RL training that harvests preemptible GPU resources. Our key insight is that rollout’s stateless and embarrassingly parallel nature aligns perfectly with preemptible and often fragmented resources. To efficiently utilize these resources despite frequent and unpredictable availability changes, RLBoost adopts a hybrid architecture with three key techniques: (1) adaptive rollout offload to dynamically adjust workloads on the reserved (on-demand) cluster, (2) pull-based weight transfer that quickly provisions newly available instances, and (3) token-level response collection and migration for efficient preemption handling and continuous load balancing. Extensive experiments show RLBoost increases training throughput by 1.51x-1.97x while improving cost efficiency by 28%-49% compared to using only on-demand GPU resources. † † ∗ Yongji Wu and Xueshen Liu contributed equally. † Work was done when Xueshen Liu interned at Google. \ddagger Our code has been merged into https://github.com/Terra-Flux/PolyRL . 1. Introduction Figure 1. Architectures for on-policy RL frameworks. Reinforcement learning (RL) post-training has become the key enabler in unlocking advanced reasoning capabilities for modern large language models (LLMs). RL not only empowers state-of-the-art LLMs like Claude 4 ( cla 2025 ) and Grok 4 ( gro 2025 ) to achieve leading performance in mathematics, coding, and tool use, but also enables smaller, more efficient models to reach or even surpass the performance of much larger LLMs on specialized tasks ( Zheng et al. 2025 ; Prabhakar et al. 2025 ; Wu et al. 2025 ; Zhou et al. 2025 ) . Unlike traditional pre-training, the RL workflow is mainly composed of two interdependent stages: rollout and training. In the rollout stage, input prompts are fed to inference engines, e.g., vLLM ( Kwon et al. 2023 ) , to generate a batch of responses. The responses are then used in the training stage to derive reward signals, compute loss, and update model weights. The updated model is subsequently transferred to the inference engines for the rollout stage of the next iteration. Existing RL frameworks can be divided into two categories: disaggregated and co-located. In disaggregated frameworks, rollout and training stages are assigned to separate sets of GPUs. They either struggle with resource under-utilization ( Figure 1 (a)) due to bubbles caused by stage dependency ( Shen et al. 2024 ; Hu et al. 2024 ) , or sacrifice model accuracy by using asynchronous (off-policy) algorithms ( Zhong et al. 2025 ; Fu et al. 2025 ; Han et al. 2025 ) to relax the stage dependency. To maximize resource utilization under the well-established synchronous (on-policy) RL algorithms, co-located RL frameworks are proposed ( Sheng et al. 2025 ; Zhong et al. 2024 ; Mei et al. 2024 ) , where training and rollout stages are on the same set of GPUs. The two stages time-share the GPUs, with each GPU alternating between rollout and training, avoiding idle GPU cycles. However, the two stages exhibit fundamentally different resource requirements. In terms of resource types, the rollout stage partitions available GPUs into multiple independent rollout instances, using tensor parallelism within each instance and requiring no communication across instances. In contrast, the training stage generally employs fully sharded data parallelism (FSDP) ( Zhao et al. 2023 ) and/or 3D parallelism across all available GPUs, involving extensive full-mesh communication between GPUs. In terms of resource quantities, the rollout stage scales efficiently by spawning more independent rollout instances and substantially benefits from allocating more GPUs than the training stage, as generation takes up to 90% of overall RL time under the co-located setting ( He et al. 2025 ) . How can we reconcile this fundamental resource tension under synchronous algorithms without compromising system efficiency or incurring prohibitive monetary costs? Public clouds and private production GPU clusters typically offer their excess capacity in the form of preemptible resources. Spot instances on public clouds provide considerable cost savings (up to 90% cheaper) ( Thorpe et al. 2023 ) , while production clusters generally have unused GPUs reserved for online workloads ( Duan et al. 2024 ; Newell et al. 2021 ) . These instances can be preempted at any time by the infrastructure provider. Moreover, these spare GPU resources often suffer from fragmentation at multiple levels, leading to significant communication overhead. At the node level, available GPUs may spread across many nodes, with each node already partially occupied ( Weng et al. 2023 ; gpu 2025 ) . At the cluster level, available nodes may be topologically scattered across different racks or pods, causing traffic to cross spine and core switches ( Rajasekaran et al. 2024 ; clu 2022 ) . These preemptible and fragmented resources, while poorly suited for training, align well with the rollout stage’s embarrassingly parallel and stateless nature. Our insight is that through a hybrid architecture, we can harvest preemptible resources for high throughput and cost-efficient RL on LLMs. Under the hybrid architecture shown in Figure 1 (c), the reserved training cluster performs both training and rollout but opportunistically outsources part of the rollout workload to available preemptible rollout instances. Still, to efficiently harvest these preemptible resources, there are several key challenges. First, how can we adapt the workloads on the training cluster to dynamic preemptible resource availability? Second, when a new preemptible instance becomes available, how can we quickly provision it with the latest model weights for it to begin rollout, while minimizing progress loss when an instance is preempted? Third, how can we balance the load across rollout instances? The output length of rollout requests in RL exhibits high non-determinism ( Yu et al. 2025 ) , which is further complicated by instance elasticity. Without careful scheduling, tail requests cause severe load concentration on a small subset of instances. To address these challenges, we propose RLBoost, a systematic solution for RL with a hybrid architecture that harvests preemptible resources. To adapt to dynamic resource availability, RLBoost employs an adaptive rollout offload mechanism. At each step, the training cluster starts from a "seeding" stage, where it is temporarily repurposed for rollout. During this stage, it pre-computes a part