Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference
Euijun Chung, Yuxiao Jia, Aaron Jezghani, Hyesoon Kim · 2026
Read Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference by Euijun Chung, Yuxiao Jia, Aaron Jezghani, Hyesoon Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22774.txt.
Description
Chung and co-authors show that CPUs, not GPUs, often slow multi-GPU language model serving. In their study, GPUs sit idle because CPUs cannot keep them fed. Signs include late kernel launches, stalled communication, and slow tokenization. The problem worsens in agent serving, where long contexts raise CPU work. Prefix cache reuse cuts GPU work at the same time. Adding CPU cores proves a cheap fix. Under moderate load, well-fed setups cut time to first token by 1.47 to 7.11 times.
Catalog record
| CREATOR | Euijun Chung, Yuxiao Jia, Aaron Jezghani, Hyesoon Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53060 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.22774