ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
Hao Kang ∗, Ziyang Li ∗, Xinyu Yang ∗, Weili Xu †, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora · 2026
Read ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System by Hao Kang ∗, Ziyang Li ∗, Xinyu Yang ∗, Weili Xu †, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13692.txt.
Description
Agentic workflows chain many language-model and tool requests, yet current systems assemble inference engines and tool orchestrators separately, scheduling each request without end-to-end knowledge of the workflow. That separation wastes KV cache and tool execution environments. ThunderAgent instead abstracts workflows as LLM Programs, giving a unified view of heterogeneous resources including KV caches, system states, and tool assets such as disk memory and network ports.
A program-aware scheduler and a tool resource manager built on that abstraction maximize cache hit rates, mitigate memory imbalances, and enable asynchronous environment preparation. Across coding, routing, and scientific discovery agents, throughput improved 1.5 to 3.6 times in serving and 1.8 to 3.9 times in RL rollout, with up to 4.2 times disk memory savings. Hao Kang leads the work with nine co-authors.
Catalog record
| CREATOR | Hao Kang ∗, Ziyang Li ∗, Xinyu Yang ∗, Weili Xu †, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-13692