DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
Yuhan Liu, Yuyang Huang Jiayi Yao Shaoting Feng Zhuohan Gu Kuntai Du Hanchen Li Yihua Cheng, Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse · 2024
Read DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving by Yuhan Liu, Yuyang Huang Jiayi Yao Shaoting Feng Zhuohan Gu Kuntai Du Hanchen Li Yihua Cheng, Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02820.txt.
Description
Abstract Compound AI systems , such as agentic systems, are an emerging trend in large-scale enterprise settings, with multiple LLMs specialized for different users, tasks, and/or roles working together. In these scenarios, different models often process inputs that share the same context prefix.
Catalog record
| CREATOR | Yuhan Liu, Yuyang Huang Jiayi Yao Shaoting Feng Zhuohan Gu Kuntai Du Hanchen Li Yihua Cheng, Junchen Jiang Shan Lu Madan Musuvathi Esha Choukse |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02820