Nesterov Method for Asynchronous Pipeline Parallel Optimization
Thalaiyasingam Ajanthan, Sameera Ramasinghe, Yan Zuo, Gil Avraham, Alexander Long · 2025
Read Nesterov Method for Asynchronous Pipeline Parallel Optimization by Thalaiyasingam Ajanthan, Sameera Ramasinghe, Yan Zuo, Gil Avraham, Alexander Long (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01099.txt.
Description
Thalaiyasingam Ajanthan and co-authors adapt a classic optimizer for pipeline parallel training of large neural networks. Splitting a model across small devices leaves gaps unless updates run out of sync. Async training fills the pipeline fully. The price is stale gradients, since weights move on before updates land.
Their fix reshapes the look-ahead step of Nesterov's method. The change absorbs the delay in the gradients. They prove the method still converges, at a sublinear rate, under fixed delay. In tests on language models up to a billion parameters, it beat other async methods. It even passed the synchronous baseline. That result makes small-device training of large models more practical.
Catalog record
| CREATOR | Thalaiyasingam Ajanthan, Sameera Ramasinghe, Yan Zuo, Gil Avraham, Alexander Long |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-01099