ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System

Hao Kang ∗, Ziyang Li ∗, Xinyu Yang ∗, Weili Xu †, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora · 2026

Read ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System by Hao Kang ∗, Ziyang Li ∗, Xinyu Yang ∗, Weili Xu †, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13692.txt.

Description

Agentic workflows chain many language-model and tool requests, yet current systems assemble inference engines and tool orchestrators separately, scheduling each request without end-to-end knowledge of the workflow. That separation wastes KV cache and tool execution environments. ThunderAgent instead abstracts workflows as LLM Programs, giving a unified view of heterogeneous resources including KV caches, system states, and tool assets such as disk memory and network ports.

A program-aware scheduler and a tool resource manager built on that abstraction maximize cache hit rates, mitigate memory imbalances, and enable asynchronous environment preparation. Across coding, routing, and scientific discovery agents, throughput improved 1.5 to 3.6 times in serving and 1.8 to 3.9 times in RL rollout, with up to 4.2 times disk memory savings. Hao Kang leads the work with nine co-authors.

Catalog record

CREATORHao Kang ∗, Ziyang Li ∗, Xinyu Yang ∗, Weili Xu †, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-13692

Browse the full library · About the Tower of Babel