Balanced and Elastic End-to-end Training of Dynamic LLMs

Mohamed Wahib, Muhammed Abdullah Soyturk, Didem Unat · 2025

Read Balanced and Elastic End-to-end Training of Dynamic LLMs by Mohamed Wahib, Muhammed Abdullah Soyturk, Didem Unat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14864.txt.

Description

Wahib, Soyturk, and Unat present DynMo, a load balancer for training dynamic large language models. Techniques that cut computation, such as mixtures of experts, pruning, and early exits, save work overall but leave some processors idle while others strain, which wrecks efficiency at scale. DynMo watches the imbalance and shifts work so all workers carry near-equal loads, which the authors prove is the best achievable reduction.

It can also gather the work onto fewer machines without slowing training, freeing hardware for other jobs. On dynamic GPT-style models it sped end-to-end training by factors from 1.17 to 4.5 depending on the technique, compared with standard static frameworks. The system runs on single multi-GPU machines and on clusters, and the authors present it as ready for practical deployment.

Catalog record

CREATORMohamed Wahib, Muhammed Abdullah Soyturk, Didem Unat
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14864

Browse the full library · About the Tower of Babel