Executing as You Generate: Hiding Execution Latency in LLM Code Generation

Zhensu Sun, Zhihao Lin, Zhi Chen, Chengran Yang, Mingyi Zhou, Li Li, David Lo · 2026

Read Executing as You Generate: Hiding Execution Latency in LLM Code Generation by Zhensu Sun, Zhihao Lin, Zhi Chen, Chengran Yang, Mingyi Zhou, Li Li, David Lo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-00491.txt.

Description

Sun and colleagues hide execution latency in language-model code generation by running code as it is written. Today's systems work in series: the model writes all the code, then an interpreter runs it, leaving each side idle in turn. Because a model emits tokens left to right and never revises earlier output, execution can start on early pieces while later tokens still arrive. The authors formalise this as a pipeline with generation, detection, and execution stages, and derive closed-form latency bounds for its speedup and operating range.

Catalog record

CREATORZhensu Sun, Zhihao Lin, Zhi Chen, Chengran Yang, Mingyi Zhou, Li Li, David Lo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56198
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.00491

Browse the full library · About the Tower of Babel