LlamaRL : A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
Meta GenAI · 2025
Read LlamaRL : A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training by Meta GenAI (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24034.txt.
Introduction
Bo Wu and co-authors present LlamaRL, a framework for reinforcement learning on very large language models, in a 2025 paper. Reinforcement learning is now the main way models gain skill after pretraining, but running it on models with hundreds of billions of weights strains both speed and memory. LlamaRL spreads the work across GPU clusters with a single controller built on native PyTorch, letting training steps and rollouts run out of sync.
The team proves formally that the asynchronous design speeds training up, then shows it in practice. During Llama 3 post-training, the system ran up to 10.7 times faster than a DeepSpeed-Chat style baseline on a 405-billion-parameter model. The gains grew with model size.
Catalog record
| CREATOR | Meta GenAI |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24034
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LlamaRL : A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training Meta GenAI Abstract Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs). In practice, due to the high demands on latency and memory, it is particularly challenging to develop an efficient RL framework that reliably manages policy models with hundreds to thousands of billions of parameters. In this paper, we present LlamaRL , a fully-distributed, asynchronous RL framework optimized for efficient training of large-scale LLMs with various model sizes (8B, 70B and 405B parameters) on GPU clusters with a handful to thousands devices. LlamaRL introduces a streamlined, single-controller architecture built entirely on native PyTorch, enabling modularity, ease of use, and seamless scalability to thousands of GPUs. We also provide a theoretical analysis of LlamaRL ’s efficiency, including a formal proof that its asynchronous design leads to strict RL speed-up. Empirically during the Llama 3 post training, by leveraging best practices such as co-located model offloading, asynchronous off-policy training, and distributed direct memory access for weight synchronization, LlamaRL achieves significant efficiency gains—up to 10.7x speedup compared to the DeepSpeed-Chat like systems on a 405B-parameter policy model. Furthermore, the efficiency advantage continues to grow with increasing model scale, demonstrating the framework’s suitability for future large-scale RL training. † † date: August 24, 2026 1 Introduction Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs) ( Ouyang et al.,, 2022 ; Jaech et al.,, 2024 ; DeepSeek-AI,, 2025 ) . In practice, due to the high demands on latency and memory, it is particularly challenging to develop an efficient RL framework that reliably manages policy models with hundreds to thousands of billions of parameters. In this paper, we present LlamaRL , a fully-distributed, asynchronous reinforcement learning (RL) framework optimized for efficient training of large-scale language models (LLMs) with different sizes (8B, 70B and 405B parameters, ( Llama Team,, 2024 ) ) on GPU clusters with a handful to thousands H100 GPUs. LlamaRL introduces a streamlined, single-controller architecture built entirely on native PyTorch, enabling modularity, ease of use, and seamless scalability to thousands of GPUs. We also provide a theoretical analysis of LlamaRL ’s efficiency, including a formal proof that its asynchronous design leads to strict RL speed-up. Empirically, by leveraging best practices such as co-located model offloading, asynchronous off-policy training, and GPU-native distributed weight synchronization via NVLink, LlamaRL achieves significant efficiency gains—up to 10.7x speedup compared to state-of-the-art systems (e.g., DeepSpeed-Chat ( Yao et al.,, 2023 ) ) on a 405B-parameter policy model. Furthermore, the efficiency advantage continues to grow with increasing model scale, demonstrating the framework’s suitability for future large-scale RL training. 1.1 Challenges of Scaling RL for LLM Started as a relatively lightweight calibration against human preferences ( Christiano et al.,, 2017 ; Ouyang et al.,, 2022 ) , RL has recently proved effective at enabling powerful capabilities such as reasoning ( Jaech et al.,, 2024 ; DeepSeek-AI,, 2025 ) , code generation ( Gehring et al.,, 2024 ; Wei et al.,, 2025 ) , among others. Therefore, an efficient, succinct and highly-scalable RL training framework would accelerate research progress and become beneficial to a bigger community. However, implementing such a training framework poses significant engineering challenges. These challenges can be summarized into three main categories: Flexibility to Support Various RL Algorithms. RL training process is inherently complex, not only due to the allocation of models and associated computations within the algorithm but also because of the diverse data communication among them. To date, a number of RL algorithms have been developed to finetune LLMs. PPO requires running a pipeline involving four models, i.e., actor, critic, reference policy and reward models, interacting in a sophisticated manner during training. Representative algorithms such as RAFT ( Dong et al.,, 2023 ) , ReMax ( Li et al.,, 2024 ) , RLOO ( Ahmadian et al.,, 2024 ) and GRPO ( Shao et al.,, 2024 ) have been developed without using the critic model. Multi-objective optimization frameworks such as CGPO ( Xu et al.,, 2024 ) has been developed to leverage multiple reward models and judges at the same time. As the complexity of these algorithms grows, it is important to build a flexible training framework which can easily extend to different number of models and schedule the corresponding data and parameter flows. Scaling to Larger Models and More GPUs. Training larger models consistently yields enhanced performance across a wide range of tasks ( Brown et al.,, 2020 ; Kaplan et al.,, 2020 ; Hoffmann et al.,, 2022 ) . Accommodating these ever-larger models typically requires intricate parallelism to handle both memory and computational constraints, it becomes more challenging in RL as various actors (e.g. policy model, reward model etc.) with varying sizes run simultaneously on large GPU clusters. Taking the Llama 3.1 models ( Llama Team,, 2024 ) as an example, training the 405B model via PPO requires: (1) A minimum of 512 NVIDIA’s H100 GPUs in total; (2) A model tensor parallelism (TP) size of 32 is used to reduce the memory footprint of model weights on each GPU; (3) A Fully Sharded Data Parallel (FSDP) optimization with a parallelism shard of 16 is used to further reduce the memory footprint ( Rajbhandari et al., 2019a, ; Zhao et al.,, 2023 ) ; (4) A smaller LLM model (e.g. 8B parameters) acts as a reward model. Therefore, a highly scalable RL framework is required. It must efficiently support diverse and large-scale model sizes, as well as flexible parallelism strategies, across different actors and GPU configurations to ensure efficient large-scale training. Insufficient GPU Usage due to Idle Bubbles. For LLMs, RL algorithms need to run response generation followed by loss computation and weight updates. Because different responses on different workers can have different lengths, taking different amount of time to process, such sequential execution often result in bubbles where some GPU workers are idle. The situation of insufficient GPU usage becomes even worse when the larger models introduce larger generation time differences causing larger bubbles. In some cases, slow data flow communication is another issue to waste the GPU resource. Implementing a RL training framework with high GPU utilization can not only shorten the fine-tuning wall-clock time, but also pave a way for the the bigger community to train a model with less GPU resources. Additionally, maintaining high GPU utilization while scaling both model size and GPU resources remains a significant challenge. 1.2 Contributions To address these challenges, we introduce LlamaRL , a fully-distributed, asynchronous RL training framework optimized for efficient, large-scale LLM training. Our key contributions are summarized below, following a brief background and related work discussion: • Simple and Modular Architecture: LlamaRL employs a streamlined single-controller architecture built entirely on native PyTorch. This design can seamlessly scale across thousands of GPUs, facilitating efficient training of large-scale LLMs (e.g., a 405B-parameter policy model). Its modular structure and intuitive control logic enable users to easily adapt and extend the framework, supporting diverse RL algorithms. We detail such designs in Section 5 . • High Efficiency with Scalable Best Practices: LlamaRL introduces a set of best practices specifically tailored for RL