SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling

Cunchi Lv, Xiao Shi, Dong Liang, Wenting Tan, Xiaofang Zhao · 2025

Read SpecInF: Exploiting Idle GPU Resources in Distributed DL Training via Speculative Inference Filling by Cunchi Lv, Xiao Shi, Dong Liang, Wenting Tan, Xiaofang Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02550.txt.

Description

Cunchi Lv and co-authors present SpecInF, a system that fills idle graphics processors with extra work during model training. Their paper appeared in 2025. Training large models leaves chips idle in bursts, because of uneven work sharing and waiting on network traffic. SpecInF parks inference jobs on the same chips as training. It watches for idle gaps and slips in online or offline inference tasks to fit.

In tests, the system raised chip use under common training setups. Offline inference output rose by as much as fourteenfold against one baseline. Online inference delays at the tail fell by 67 percent against another baseline. Training speed for the main job stayed protected throughout. The authors argue that data centers can win large amounts of free inference capacity from hardware they already own.

Catalog record

CREATORCunchi Lv, Xiao Shi, Dong Liang, Wenting Tan, Xiaofang Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-02550

Browse the full library · About the Tower of Babel