A Policy-Driven Runtime Layer for Agentic LLM Serving Conference: ; ; CCS: Computer systems organization Architectures CCS: Computing methodologies Machine learning

Rui Zhang, Chaeeun Kim, Liting Hu · 2026

Read A Policy-Driven Runtime Layer for Agentic LLM Serving Conference: ; ; CCS: Computer systems organization Architectures CCS: Computing methodologies Machine learning by Rui Zhang, Chaeeun Kim, Liting Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27744.txt.

Description

Rui Zhang, Chaeeun Kim, and Liting Hu insert a policy-driven runtime layer between agent frameworks and serving engines for multi-agent language models. The layer exposes observe, score, predict, and act primitives sharing agent identity as the common coordinate, hosting nine concrete policies. Validated on KV caching across sessions as CacheScout, it lifts cache hit rate by 13 to 37 points and cuts time to first token by 12 to 29 percent.

Catalog record

CREATORRui Zhang, Chaeeun Kim, Liting Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23873
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27744

Browse the full library · About the Tower of Babel