Autellix : An Efficient Serving Engine for LLM Agents as General Programs
Michael Luo Xiaoxiang Shi Colin Cai Tianjun Zhang Justin Wong, Yichuan Wang Chi Wang Yanping Huang Zhifeng Chen Joseph E. Gonzalez Ion Stoica · 2025
Read Autellix : An Efficient Serving Engine for LLM Agents as General Programs by Michael Luo Xiaoxiang Shi Colin Cai Tianjun Zhang Justin Wong, Yichuan Wang Chi Wang Yanping Huang Zhifeng Chen Joseph E. Gonzalez Ion Stoica (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13965.txt.
Description
Michael Luo and ten co-authors speed up the serving of AI agent programs, in this 2025 paper. Chatbots make one model call at a time, but agent programs chain many calls, exploring and reasoning step by step. Standard serving engines treat each call as a stranger, so programs stall behind each other's calls in long queues.
Their engine, Autellix, treats the whole program as the unit to schedule. It intercepts calls and tags them with program context. Two scheduling algorithms, for single-threaded and distributed programs, let calls jump the queue or pause based on how much of their program is already done. Across several models and agent workloads, Autellix raised program throughput four to fifteen fold at the same latency, against a leading rival system. The authors argue program-level scheduling is the right frame for the agent era.
Catalog record
| CREATOR | Michael Luo Xiaoxiang Shi Colin Cai Tianjun Zhang Justin Wong, Yichuan Wang Chi Wang Yanping Huang Zhifeng Chen Joseph E. Gonzalez Ion Stoica |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13965