Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning in LLMs
Han Zhang, Ruibin Zheng, Zexuan Yi, Hanyang Peng, Hui Wang, Yue Yu · 2025
Read Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning in LLMs by Han Zhang, Ruibin Zheng, Zexuan Yi, Hanyang Peng, Hui Wang, Yue Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17850.txt.
Introduction
Han Zhang and a large team tackle training of large models across sites spread far apart, in a paper from 2025. When learning and sampling sit in one place, slow links and mixed gear can make training jumpy and weak.
Their design splits learning from sampling, and their method, called GEPO, weighs groups of samples in a way that cuts swings in the weights. In tests, scores fell only a small amount even with long delays. The gap from best to last run shrank a great deal next to a rival method, while scores stayed the highest in the tests.
Catalog record
| CREATOR | Han Zhang, Ruibin Zheng, Zexuan Yi, Hanyang Peng, Hui Wang, Yue Yu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-17850
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learning in LLMs Han Zhang, Ruibin Zheng, Zexuan Yi, Hanyang Peng, Hui Wang, Yue Yu Abstract As the scale of single-center computing nodes approaches physical limits constrained by power capacity, decentralized distributed training is emerging as the predominant paradigm for future development. Reinforcement Learning (RL)-driven post-training, a critical pathway for enhancing the reasoning capabilities of Large Language Models (LLMs), typically alternates between homologous sampling (Rollout) and learning tasks in isomorphic environments. This approach struggles to adapt to the emerging demand for large-scale, decentralized training leveraging geographically distributed, asynchronous heterogeneous computing resources. To address this, we propose HeteroRL , an asynchronous RL architecture specifically designed for heterogeneous environments to efficient LLM mathematical reasoning training. HeteroRL decouples the Rollout sampler from the parameter learner, enabling deployment across distinct computing nodes and tolerating temporal delays induced by network transmission. We identify that high network latency impacts the RL training process primarily through increased KL divergence between the sampler and learner, ultimately causing importance sampling to fail due to excessive variance. However, internet network latency is inherently unpredictable. To mitigate this, we propose Group Expectation Policy Optimization (GEPO) , which reduces the variance of importance weights by refining the standard importance sampling calculation, thereby maintaining training stability and enhancing performance. Theoretically, we demonstrate that this method can exponentially reduce the variance of importance weights. Experiments on RL training with heterogeneous computing resources show that GEPO achieves superior training stability compared to popular methods like GRPO on mathematical reasoning benchmarks. Under extreme network delays of up to 1800 seconds, the performance degradation remains within 3% compared to single-center (synchronous) training, validating its significant potential for efficient decentralized distributed RL training in heterogeneous computing networks. Keywords: Decentralized Training, Reinforcement Learning, Heterogeneous Computing, Large Language Models, Mathematical Reasoning, Importance Sampling Figure 1: Left: GEPO improves upon GRPO and GSPO by employing sample-level importance sampling weights and group expectation smoothing to enhance training stability. Right: In both zero-delay (online) and high-delay (up to 1800 seconds) heterogeneous reinforcement learning scenarios, GEPO demonstrates superior stability and better evaluation performance (MATH-500). 1 Introduction In recent years, Large-scale Language Models (LLMs) ( Dubey et al. 2024 ; Yang et al. 2025 ; Brown et al. 2020 ) have achieved remarkable progress through pre-training on massive text corpora. However, as high-quality pre-training data becomes increasingly scarce, the paradigm of solely relying on scaling up model size and data volume is encountering diminishing returns ( Kaplan et al. 2020 ) , with limited room for further performance improvement. To further unlock the potential of LLMs, particularly in enhancing their capabilities on complex tasks such as mathematical reasoning Shao et al. 2024 , Reinforcement Learning (RL) ( Bai et al. 2022 ; Jaques et al. 2020 ; Zhou & Xu 2020 ; Kreutzer et al. 2018 ; Stiennon et al. 2020 ) has been widely adopted as an effective post-training method. RL allows the model to generate reasoning processes (e.g., chain-of-thought ( Kojima et al. 2022 ) ) through interaction with an environment and optimize based on reward signals ( Shao et al. 2024 ) (e.g., the correctness of answers), thereby iteratively improving its reasoning abilities. Despite the significant potential of RL in enhancing LLM reasoning, its large-scale distributed training faces serious systemic challenges. On the one hand, traditional synchronous RL frameworks require strict alternation between rollout generation and policy/critic updates ( Han et al. 2025 ; Ji et al. 2023 ; Gao et al. 2023 ; Warnell et al. 2018 ) . This leads to significant idle time for computational resources (e.g., GPUs) while waiting for the completion of the longest sequences when processing long reasoning outputs, severely constraining training efficiency ( Fu et al. 2025 ) . On the other hand, with the explosive growth in model size and data requirements, the computational power of a single node is no longer sufficient. Leveraging geographically distributed heterogeneous computing resources to form computing networks for decentralized training has become an inevitable trend ( Team et al. 2025 ) . In this scenario, network latency 1 1 1 In this paper, the terms ’delay’ and ’latency’ are used interchangeably. between different nodes is inevitable, resulting in a temporal gap (i.e., delay or staleness) between the policy version used by the sampler and the latest policy version on the learner. Most existing RL frameworks are tightly coupled with homogeneous, low-latency computing environments and struggle to effectively address the challenges posed by such heterogeneity and latency, limiting their practical application in large-scale distributed deployments. To address the above challenges, we propose HeteroRL (Heterogeneous Reinforcement Learning), a reinforcement learning framework specifically designed for asynchronous and heterogeneous environments, aiming to achieve efficient and robust mathematical reasoning training for large language models. The core idea of HeteroRL is to decouple the two computationally intensive yet distinct tasks in the RL pipeline—rollout sampling and parameter learning—and deploy them on independent computing nodes that may have different computational capabilities. The sampler is responsible for continuously generating reasoning trajectories, while the learner asynchronously updates the model parameters based on the collected trajectory data. Both processes run continuously without waiting for each other, communicating via dedicated networks or the internet to transfer model parameters and sampled data at low frequencies or under high latency. The heterogeneous reinforcement learning scenario differs from the traditional online reinforcement learning scenario, as shown in Table 1 : Table 1: Core Characteristics of Online, Offline, and Heterogeneous Reinforcement Learning Paradigms Aspect Online RL Offline RL Heterogeneous RL Data Generation Online interaction: Data generated instantly by the current policy . Fixed, pre-collected dataset: Uses static data collected by some (unknown) behavior policy. Delayed interaction: Data generated by historical policy versions (due to unpredictable network delay ). Policy Version during Data Generation Always current : Requires strict synchronization with the learning policy. Fixed : The behavior policy is fixed and inherent to the dataset. Dynamically stale : Staleness determined by unpredictable network delay (core characteristic) . System Architecture Tightly-coupled: Actor (environment interaction) and Learner (parameter update) typically co-located or on a low-latency network. Single-machine or simple distributed: No real-time interaction required; training is data-driven. Geographically decoupled: Actor and Learner separated by a high-latency network , tolerant to delays. Handling of System Latency Treated as failure: Requires synchronization; latency causes resource idling and training stalls. Not applicable: Training process has no real-time interaction . Algorithmic compensation : Employs corrective techniques (e.g., importance sampling) to mitigate latency effects. Core Challenge Exploration-exploitation trade-off during learning. *Resource utilization* under sequential tasks. Distributional shift and limited data coverage . Algorithmic stability under dynamic delays : High/variable latency causes importance weight variance explosion* . Note: *High variance in importance weights is a critical challenge identified in Heterogeneous RL due to dynamic delays. To address the challenge of policy version mismatch caused by network latency in heterogeneous distributed environments, we conduct an in-depth investigation and propose a novel reinforcement learning algorithm tailored for such settings. By designing a carefully crafted objective function, our method effectively balances training efficiency, convergence, and final model performance under dynamically varying communication delays. Our findings reveal a particularly noteworthy insight: moderate temporal asynchrony does not necessarily harm model training—in fact, within a certain range, it can