Balanced and Elastic End-to-end Training of Dynamic LLMs
Mohamed Wahib, Muhammed Abdullah Soyturk, Didem Unat · 2025
Read Balanced and Elastic End-to-end Training of Dynamic LLMs by Mohamed Wahib, Muhammed Abdullah Soyturk, Didem Unat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14864.txt.
Description
Wahib, Soyturk, and Unat present DynMo, a load balancer for training dynamic large language models. Techniques that cut computation, such as mixtures of experts, pruning, and early exits, save work overall but leave some processors idle while others strain, which wrecks efficiency at scale. DynMo watches the imbalance and shifts work so all workers carry near-equal loads, which the authors prove is the best achievable reduction.
It can also gather the work onto fewer machines without slowing training, freeing hardware for other jobs. On dynamic GPT-style models it sped end-to-end training by factors from 1.17 to 4.5 depending on the technique, compared with standard static frameworks. The system runs on single multi-GPU machines and on clusters, and the authors present it as ready for practical deployment.
Catalog record
| CREATOR | Mohamed Wahib, Muhammed Abdullah Soyturk, Didem Unat |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-14864