SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling
Cunchi Lv, Xiao Shi, Dong Liang, Wenting Tan, Xiaofang Zhao · 2025
Read SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling by Cunchi Lv, Xiao Shi, Dong Liang, Wenting Tan, Xiaofang Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02550.txt.
Description
Cunchi Lv and co-authors present SpecInF, a system that fills idle graphics processors with extra work during model training. Their paper appeared in 2025. Training large models leaves chips idle in bursts, because of uneven work sharing and waiting on network traffic. SpecInF parks inference jobs on the same chips as training. It watches for idle gaps and slips in online or offline inference tasks to fit.
In tests, the system raised chip use under common training setups. Offline inference output rose by as much as fourteenfold against one baseline. Online inference delays at the tail fell by 67 percent against another baseline. Training speed for the main job stayed protected throughout. The authors argue that data centers can win large amounts of free inference capacity from hardware they already own.
Catalog record
| CREATOR | Cunchi Lv, Xiao Shi, Dong Liang, Wenting Tan, Xiaofang Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-02550