Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference

Euijun Chung, Yuxiao Jia, Aaron Jezghani, Hyesoon Kim · 2026

Read Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference by Euijun Chung, Yuxiao Jia, Aaron Jezghani, Hyesoon Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-22774.txt.

Description

Chung and co-authors show that CPUs, not GPUs, often slow multi-GPU language model serving. In their study, GPUs sit idle because CPUs cannot keep them fed. Signs include late kernel launches, stalled communication, and slow tokenization. The problem worsens in agent serving, where long contexts raise CPU work. Prefix cache reuse cuts GPU work at the same time. Adding CPU cores proves a cheap fix. Under moderate load, well-fed setups cut time to first token by 1.47 to 7.11 times.

Catalog record

CREATOREuijun Chung, Yuxiao Jia, Aaron Jezghani, Hyesoon Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53060
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.22774

Browse the full library · About the Tower of Babel