Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start
Xueshen Liu 1 , ∗ {}^{\text{1},*} Yongji Wu 2 , ∗ , † {}^{\text{2},*,\text{\textdagger}} Yuncheng Yao 3 {}^{\text{3}} Danyang Zhuo 3 {}^{\text{3}} Ion Stoica 2 {}^{\text{2}} Z. Morley Mao 1 {}^{\text{1}} 1 {}^{\text{1}} University of Michigan 2 {}^{\text{2}} UC Berkeley 3 {}^{\text{3}} Duke · 2026
Read Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start by Xueshen Liu 1 , ∗ {}^{\text{1},*} Yongji Wu 2 , ∗ , † {}^{\text{2},*,\text{\textdagger}} Yuncheng Yao 3 {}^{\text{3}} Danyang Zhuo 3 {}^{\text{3}} Ion Stoica 2 {}^{\text{2}} Z. Morley Mao 1 {}^{\text{1}} 1 {}^{\text{1}} University of Michigan 2 {}^{\text{2}} UC Berkeley 3 {}^{\text{3}} Duke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06664.txt.
Description
Liu and five colleagues attack the cold-start bottleneck in large-model serving with Foundry. CUDA graph capture still takes tens of seconds to minutes, dwarfing weight loading, yet graphs resist naive saving, embedding device addresses and lazily loaded code. Foundry saves both graph shape and execution context offline, then rebuilds working graphs online cheaply, even turning single-GPU captures into multi-GPU templates. Across models up to 235B, cold starts fall by up to 99 percent, with Qwen3-235B-A22B falling from 10 minutes to under 4 seconds.
Catalog record
| CREATOR | Xueshen Liu 1 , ∗ {}^{\text{1},*} Yongji Wu 2 , ∗ , † {}^{\text{2},*,\text{\textdagger}} Yuncheng Yao 3 {}^{\text{3}} Danyang Zhuo 3 {}^{\text{3}} Ion Stoica 2 {}^{\text{2}} Z. Morley Mao 1 {}^{\text{1}} 1 {}^{\text{1}} University of Michigan 2 {}^{\text{2}} UC Berkeley 3 {}^{\text{3}} Duke |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60570 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.06664