One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev, Samuel Horváth · 2026
Read One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining by Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev, Samuel Horváth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30634.txt.
Description
Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev and Samuel Horváth revisit a feared bottleneck in training large language models. Pipeline parallelism splits a model across processors in stages, but synchronous schedules leave processors idle in pipeline bubbles. Asynchronous schedules remove the bubbles at the cost of stale gradients. The PipeDream-2BW schedule keeps the delay at exactly one step, yet few use it, fearing instability.
The team show the fear depends on the optimiser, the rule that updates model weights. AdamW, dominant when the schedule appeared, degrades badly under one-step delay. The newer Muon optimiser stays robust. The authors add an optimiser-agnostic correction inspired by error feedback to close the remaining gap, and prove convergence for Muon with and without it. Tests on models up to ten billion parameters matched synchronous training, supporting asynchronous pipelines at scale.
Catalog record
| CREATOR | Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev, Samuel Horváth |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 109 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30634