Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start

Xueshen Liu 1 , ∗ {}^{\text{1},*} Yongji Wu 2 , ∗ , † {}^{\text{2},*,\text{\textdagger}} Yuncheng Yao 3 {}^{\text{3}} Danyang Zhuo 3 {}^{\text{3}} Ion Stoica 2 {}^{\text{2}} Z. Morley Mao 1 {}^{\text{1}} 1 {}^{\text{1}} University of Michigan 2 {}^{\text{2}} UC Berkeley 3 {}^{\text{3}} Duke · 2026

Read Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start by Xueshen Liu 1 , ∗ {}^{\text{1},*} Yongji Wu 2 , ∗ , † {}^{\text{2},*,\text{\textdagger}} Yuncheng Yao 3 {}^{\text{3}} Danyang Zhuo 3 {}^{\text{3}} Ion Stoica 2 {}^{\text{2}} Z. Morley Mao 1 {}^{\text{1}} 1 {}^{\text{1}} University of Michigan 2 {}^{\text{2}} UC Berkeley 3 {}^{\text{3}} Duke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06664.txt.

Description

Liu and five colleagues attack the cold-start bottleneck in large-model serving with Foundry. CUDA graph capture still takes tens of seconds to minutes, dwarfing weight loading, yet graphs resist naive saving, embedding device addresses and lazily loaded code. Foundry saves both graph shape and execution context offline, then rebuilds working graphs online cheaply, even turning single-GPU captures into multi-GPU templates. Across models up to 235B, cold starts fall by up to 99 percent, with Qwen3-235B-A22B falling from 10 minutes to under 4 seconds.

Catalog record

CREATORXueshen Liu 1 , ∗ {}^{\text{1},*} Yongji Wu 2 , ∗ , † {}^{\text{2},*,\text{\textdagger}} Yuncheng Yao 3 {}^{\text{3}} Danyang Zhuo 3 {}^{\text{3}} Ion Stoica 2 {}^{\text{2}} Z. Morley Mao 1 {}^{\text{1}} 1 {}^{\text{1}} University of Michigan 2 {}^{\text{2}} UC Berkeley 3 {}^{\text{3}} Duke
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60570
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06664

Browse the full library · About the Tower of Babel