One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining

Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev, Samuel Horváth · 2026

Read One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining by Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev, Samuel Horváth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30634.txt.

Description

Philip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev and Samuel Horváth revisit a feared bottleneck in training large language models. Pipeline parallelism splits a model across processors in stages, but synchronous schedules leave processors idle in pipeline bubbles. Asynchronous schedules remove the bubbles at the cost of stale gradients. The PipeDream-2BW schedule keeps the delay at exactly one step, yet few use it, fearing instability.

The team show the fear depends on the optimiser, the rule that updates model weights. AdamW, dominant when the schedule appeared, degrades badly under one-step delay. The newer Muon optimiser stays robust. The authors add an optimiser-agnostic correction inspired by error feedback to close the remaining gap, and prove convergence for Muon with and without it. Tests on models up to ten billion parameters matched synchronous training, supporting asynchronous pipelines at scale.

Catalog record

CREATORPhilip Zmushko, Egor Petrov, Nursultan Abdullaev, Mikhail Khrushchev, Samuel Horváth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE109 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30634

Browse the full library · About the Tower of Babel