Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management
Thanh Son Phung, Douglas Thain · 2025
Read Scaling Up Throughput-oriented LLM Inference Applications on Heterogeneous Opportunistic GPU Clusters with Pervasive Context Management by Thanh Son Phung, Douglas Thain (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13201.txt.
Description
Thanh Son Phung and Douglas Thain pool spare GPUs in 2025 for throughput-first language model jobs. Demand outstrips cluster supply, and static holds force queues or costly buys. Not all jobs need low delay, so spare capacity can serve them if context moves well.
Shared context reuse is the core fix. Policies let work hop across spare nodes without loss. A test job cut its run time by near all versus static holds. The gain shows spare pools can rival owned clusters for batch jobs. Context care, not raw cards, is the bottleneck named.
Catalog record
| CREATOR | Thanh Son Phung, Douglas Thain |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-13201