Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training

Brian R. Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, Bhavya Kailkhura · 2025

Read Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training by Brian R. Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, Bhavya Kailkhura (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18929.txt.

Introduction

Brian Bartoldson and co-authors decouple exploration from learning to speed up reinforcement learning post-training of language models. Standard on-policy training forces the model to learn only from its freshest attempts, wasting the parallel power of many actors generating examples at once. Off-policy data, though, poison classic objectives.

Their method, Trajectory Balance with Asynchrony, learns soundly from the mixed replay stream that asynchronous actors produce. Across math solving, preference tuning, and automated red-teaming, and on models from small to seven billion parameters, it beat strong baselines in speed and quality. Post-training at scale is a logistics problem as much as a math one.

Catalog record

CREATORBrian R. Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, Bhavya Kailkhura
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18929

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training Brian R. Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, Bhavya Kailkhura Abstract Reinforcement learning (RL) is a critical component of large language model (LLM) post-training. However, existing on-policy algorithms used for post-training are inherently incompatible with the use of experience replay buffers, which can be populated scalably by distributed off-policy actors to enhance exploration as compute increases. We propose efficiently obtaining this benefit of replay buffers via Trajectory Balance with Asynchrony (TBA), a massively scalable LLM RL system. In contrast to existing approaches, TBA uses a larger fraction of compute on search, constantly generating off-policy data for a central replay buffer. A training node simultaneously samples data from this buffer based on reward or recency to update the policy using Trajectory Balance (TB), a diversity-seeking RL objective introduced for GFlowNets. TBA offers three key advantages: (1) decoupled training and search, speeding up training wall-clock time by 4x or more; (2) improved diversity through large-scale off-policy sampling; and (3) scalable search for sparse reward settings. On mathematical reasoning, preference-tuning, and automated red-teaming (diverse and representative post-training tasks), TBA produces speed and performance improvements over strong baselines. Figure 1: TBA performs rapid, scalable exploration of model responses, improving RL efficiency on the GSM8K mathematical reasoning task . All plotted points use 4xA100 GPUs (or comparable L40S GPUs). DPO and RLOO baselines taken from Noukhovitch et al. 2025 , PPO and VinePPO baselines taken from Kazemnejad et al. 2024 . The baseline model is the SFTed RhoMath-1B ( Lin et al. 2024 ) model, which obtains 40.3% accuracy after SFT and before RL. Appendix A has details. 1 Introduction Post-training through reinforcement learning (RL) is a critical step in enhancing large language models (LLMs), aligning them with human preferences and improving their reasoning abilities ( Christiano et al. 2017 ) . However, widely used RL algorithms such as Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) and REINFORCE Leave-One-Out (RLOO) ( Ahmadian et al. 2024 ) suffer from a fundamental limitation: they are on-policy , meaning that data generation and policy updates occur sequentially. This dependence creates bottlenecks that reduce resource utilization. Moreover, the benefit of scaling on-policy data generation may be limited ( Hou et al. 2024 ) . We introduce Trajectory Balance with Asynchrony (TBA) , a distributed RL framework designed to efficiently and scalably leverage compute for LLM post-training. TBA decouples data generation from policy updates using an off-policy training objective based on Trajectory Balance (TB) ( Malkin et al. 2022a ) . The framework uses multiple searcher nodes that independently generate diverse trajectories and store them in a central replay buffer , while a single trainer node asynchronously samples from this buffer to update the policy. By removing the dependency between data generation and training , TBA ensures high resource utilization and facilitates scalable search. By computing model updates with large, efficiently-generated off-policy response sets, the TB objective leverages response scaling to produce better models than popular RL baseline methods. TBA offers three key advantages over existing RL-based LLM post-training approaches: 1) Asynchrony enables massive parallelization , significantly reducing training wall-clock time as shown in Figures 1 and 3 . 2) Diverse off-policy sampling from a replay buffer improves exploration and prevents mode collapse. 3) Scalable search capabilities make TBA particularly effective in sparse reward settings , such as automated red-teaming. We validate TBA on mathematical reasoning , preference-tuning , and automated red-teaming . Our results show that TBA achieves performance comparable to or better than existing methods while significantly improving training speed. Our key contributions are summarized as follows: • We introduce TBA , a novel distributed RL framework for LLM post-training. • We decouple data generation and policy updates , improving training speed and scalability. • We demonstrate the effectiveness of trajectory balance for LLM post-training , exploiting its ability to efficiently leverage large-scale off-policy data. • We demonstrate significant speedups (4x or more) in RL for mathematical reasoning, preference-tuning, and automated red-teaming. By enabling high-quality and fast off-policy post-training, TBA contributes to the broader goal of scalable and effective LLM alignment , ensuring that large models can be refined more efficiently for real-world deployment. 2 Related Work RL fine-tuning of language models Reinforcement Learning (RL) has been an integral component for training LLMs ( Google Gemini Team 2024 ; OpenAI 2023 ) . In particular, RL has became the de facto approach for aligning language models with human preferences Christiano et al. 2017 ; Ziegler et al. 2019 ; Stiennon et al. 2020 ; Ouyang et al. 2022 . Much of this work relies on Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) , an on-policy RL algorithm which has become a default choice for fine-tuning LLMs due to it’s strong performance across different setups. Aside from PPO, other on-policy objectives such as REINFORCE ( Ahmadian et al. 2024 ) and variants like GRPO ( Shao et al. 2024 ) and VinePPO ( Kazemnejad et al. 2024 ) have also been studied in the context of language models. An alternative to PPO-based fine-tuning is rejection sampling fine-tuning, inspired by the best-of- n inference approach proposed by Nakano et al. 2021 . Recent work by Dong et al. 2023 ; Gulcehre et al. 2023 , and Wang et al. 2024 extends this concept by generating n candidate responses for each prompt, ranking them with a learned reward function, and fine-tuning the model based on the highest-ranking responses. On the other hand, direct preference learning approaches ( Rafailov et al. 2023 ; Azar et al. 2024 ; Tang et al. 2024 ) skip reward modeling entirely and train language models to directly optimize responses under a preference model. Finally, Hu et al. 2024 introduced GFlowNet fine-tuning, leveraging off-policy GFlowNet algorithms for fine-tuning language models, which we build upon in this work. Asynchronous distributed RL Distributed RL spreads actors/searchers, learners, and environments across a collection of computing resources. Asynchronous distributed RL does this such that searcher and trainer processes do not necessarily share the same weights, which can significantly improve training speed ( Nair et al. 2015 ; Wang et al. 2025 ) and facilitate RL in complex, high-dimensional domains ( Hessel et al. 2021 ; Huang et al. 2023 ; Horgan et al. 2018 ) . A foundational method in this area is Asynchronous Advantage Actor-Critic (A3C) ( Mnih 2016 ) . In A3C, multiple parallel workers asynchronously interact with the environment and communicate gradients to a central node. Our approach to async distributed RL more closely resembles the Importance-Weighted Actor-Learner Architecture (IMPALA) method ( Espeholt et al. 2018 ) , which communicates experience trajectories (state, action, and reward tuples) to the central node. Automated red-teaming Through adversarial interactions, LLM red-teaming clarifies the robustness and risks of a target LLM. Automating the generation of these adversarial scenarios, automated red-teaming frameworks can help uncover vulnerabilities, biases, and unintended behaviors in models more quickly, enabling preemptive mitigation before deployment ( Wei et al. 2023 ; Ji et al. 2024 ) . Perez et al. 2022 proposed training language models using RL to discover

Read the complete text