MLP-Offload : Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis; November 16–21, 2025; St Louis, MO, USA The International Conference for High Performance Comp
Avinash Maurya, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae · 2025
Read MLP-Offload : Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall Conference: The International Conference for High Performance Computing, Networking, Storage and Analysis; November 16–21, 2025; St Louis, MO, USA The International Conference for High Performance Comp by Avinash Maurya, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02480.txt.
Description
Avinash Maurya, M. Mustafa Rafique, Franck Cappello, and Bogdan Nicolae present a 2025 systems paper on training language models too large for combined GPU memory. Overflow data spills to host memory or disk, but moving it back and forth slows every training step.
Their MLP-Offload engine spreads optimiser state, the bookkeeping data of training, across several storage tiers and paths. Design choices follow three observations: update-phase transfers dominate step time, remote storage bandwidth sits idle, and competing transfers worsen congestion. Cache-aware placement plus controlled concurrency ease those bottlenecks. On models up to hundreds of billions of parameters, iterations ran about two and a half times faster than leading training runtimes, easing the memory wall for modest clusters.
Catalog record
| CREATOR | Avinash Maurya, M. Mustafa Rafique, Franck Cappello, Bogdan Nicolae |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02480