DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving

Yuhan Liu, Yuyang Huang Jiayi Yao Shaoting Feng Zhuohan Gu Kuntai Du Hanchen Li Yihua Cheng, Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse · 2024

Read DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving by Yuhan Liu, Yuyang Huang Jiayi Yao Shaoting Feng Zhuohan Gu Kuntai Du Hanchen Li Yihua Cheng, Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02820.txt.

Description

Abstract Compound AI systems , such as agentic systems, are an emerging trend in large-scale enterprise settings, with multiple LLMs specialized for different users, tasks, and/or roles working together. In these scenarios, different models often process inputs that share the same context prefix.

Catalog record

CREATORYuhan Liu, Yuyang Huang Jiayi Yao Shaoting Feng Zhuohan Gu Kuntai Du Hanchen Li Yihua Cheng, Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02820

Browse the full library · About the Tower of Babel