Align and Filter: Improving Performance in Asynchronous On-Policy RL

Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, Glen Berseth · 2026

Read Align and Filter: Improving Performance in Asynchronous On-Policy RL by Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, Glen Berseth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01365.txt.

Introduction

Honari and five colleagues tackle policy lag in async on-policy RL, the gap between the acting policy making data and the learning policy updated. Spread training and frequent updates speed learning but worsen this lag, blocking scaling to larger problems. They name policy lag sources from distributed learning and high update pace, then propose total Variation-based Advantage aligned Constrained policy Optimization, a usable fix for policy lag. Tests show better toughness to policy lag in old RL tasks and a modern RL task for language model math thought.

Catalog record

CREATORHomayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, Glen Berseth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63396
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01365

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Align and Filter: Improving Performance in Asynchronous On-Policy RL Homayoun Honari, Roger Creus Castanyer, Michael Przystupa, Michael Noukhovitch, Pablo Samuel Castro, Glen Berseth Abstract Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: policy lag , which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose total Variation-based Advantage aligned Constrained policy Optimization (VACO) as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task. 1 Introduction Distributed reinforcement learning (RL) provides highly optimized and fast training pipelines by updating policies and collecting data simultaneously by using decentralized compute systems ( Langer et al. 2020 ) . In this setup, there are multiple compute nodes separately generating the data which is then sent to a learner node for processing. For example, multiple robots can run a policy to perform manipulation or locomotion tasks locally and transmit their collected experience to a central compute node to improve the policy ( Gu et al. 2017 ; Rudin et al. 2022 ) . One of the most common difficulties in this setup is the repeated failures and delays in communication which causes the data distribution to differ from the learning policy distribution ( Yuan & Mahmood 2022 ) . This data distribution mismatch violates a key assumption in on-policy deep RL algorithms which assumes the behavioral and the target policy are the same ( Sutton et al. 1998 ) . On-policy methods operate by generating data using environment rollouts with the current policy, and making conservative updates to keep close to the data distribution ( Schulman et al. 2015a ) . For example, Proximal Policy Optimization (PPO) and its variants quantify the closeness of the distributions using the KL divergence ( Schulman et al. 2017 ; Wang et al. 2020 ) . Hence, the policy is optimized to increase reward, while bounding the divergence from the behavior policy. This allows a policy to make more mini-batch updates on the data to improve the policy’s performance, but it widens the gap between the data distribution and the most up-to-date policy. This policy gap can lead to severe performance degradation or even full policy collapse ( Nikishin et al. 2022 ) . This limitation has previously been defined as policy lag in the literature and has been known to hinder policy improvement ( Babaeizadeh et al. 2016 ; Petrenko et al. 2020 ) . In this paper, we provide a categorization of the sources of policy lag: backward policy lag , which arises from the initial mismatch between the behavior and learning policies, and forward policy lag , which accumulates as gradient updates cause the learning policy to diverge from the data distribution. This forward lag creates a critical trade-off: while more updates per batch of data may improve performance, they also increase the risk of policy degradation. We demonstrate that Total Variation (TV) divergence provides an effective tool for quantifying both sources of policy lag. We propose Total Variation-based Advantage-aligned Constrained policy Optimization (VACO) to address the forward and backward policy lag which inherently exists in asynchronous RL. VACO is based on two main ideas: advantage realignment and TV divergence-based filtering . We use advantage realignment to estimate the advantage function of the learning policy from off-policy data which allows us to mitigate the problem of backward policy lag. To control the forward policy lag that can arise from the policy optimization process, we then use TV divergence to filter data points in each minibatch that would amplify the effect of forward policy lag. We find that our filtering strategy is effective in maintaining a certain threshold on the value of the expected TV divergence without the additional complexity for the choice of hyperparameters for constraint satisfaction. To study the effect of backward policy lag in various on-policy RL algorithms, we present a simulated asynchronous framework ( Howes et al. 2025 ) for RL. We use this framework to simulate highly parallelized environments in various MuJoCo ( Todorov et al. 2012 ) robotic tasks. We find that VACO offers better robustness to backward policy lag compared with PPO ( Schulman et al. 2017 ) , one of the most commonly used on-policy algorithms. We then investigate the forward policy lag present in finetuning large language models (LLMs) with RL towards math reasoning. We apply VACO to GRPO ( Shao et al. 2024 ) , the standard method in RL for LLMs, and find strong improvements in robustness to forward lags that are commonly present in these setups. In summary, we first provide a theoretical analysis of policy lag in asynchronous RL, categorizing its distinct sources. Building on this analysis, we propose VACO, a policy optimization algorithm designed for improved robustness to both forward and backward lag. We then empirically validate our method in two carefully constructed setups: highly parallelized robotics tasks and reasoning for LLMs. In both scenarios, we clearly improve robustness to policy lag compared to strong, standard RL baselines. 2 Related Work Our work investigates issues in the learning dynamics of on-policy RL algorithms when using data that arrives asynchronously to the learner agent. One of the key assumptions of on-policy methods is that data is collected with the current policy, making it challenging to reuse data or learn from off-policy samples ( Sutton et al. 1998 ) . A standard mechanism to alleviate this issue consists of defining a trust region constraint to update policies conservatively, allowing multiple updates on a given batch of data ( Schulman et al. 2015a ) . Other approximation approaches have been previously studied to mitigate the need for second-order optimizers when optimizing the policy ( Schulman et al. 2015a ; Schulman et al. 2017 ; Wang et al. 2020 ; Xie et al. 2025 ; Achiam et al. 2017b ; Vuong et al. 2019 ) . Off-policy variations have been proposed to introduce an adaptive change in the importance of the trust region regularization terms in the objective function ( Meng et al. 2023 ; Fakoor et al. 2020 ; Queeney et al. 2021 ) . Our research builds on the the TV divergence metric previously proposed to analyze the changes in data distribution in on-policy learning ( Schulman et al. 2015a ; Achiam et al. 2017a ) . Despite recent advances, on-policy methods typically require large data batches for stable optimization ( Andrychowicz et al. ; Singla et al. 2024 ) . Distributed learning architectures address this by parallelizing data collection and policy updates ( Liu et al. 2024 ; Czech 2021 ; Langer et al. 2020 ) . In these setups, a central policy either sends its weights to parallel actors for data collection ( Nair et al. 2015 ; Babaeizadeh et al. 2016 ; Mnih et al. 2016 ) or

Read the complete text