IC-Cache: Efficient Large Language Model Serving via In-context Caching

Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M. Levy, David E. Culler · 2025

Read IC-Cache: Efficient Large Language Model Serving via In-context Caching by Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M. Levy, David E. Culler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12689.txt.

Introduction

Yifan Yu and co-authors present a caching system that lets small language models borrow strength from past answers of larger models, in a paper from 2025. Serving large models is costly and slow, yet most user requests resemble earlier ones. Simply reusing old answers, though, drops quality sharply.

Their system picks similar, high value past pairs and places them in the prompt as examples. The small model can then imitate, and at times beat, larger models on mixed skills. Requests are routed across models by quality and load, and cached examples are refined offline. Tests on millions of realistic requests raised throughput several fold and cut delay, with no loss in answer quality.

Catalog record

CREATORYifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M. Levy, David E. Culler
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-12689

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IC-Cache: Efficient Large Language Model Serving via In-context Caching Yifan Yu, Yu Gan, Nikhil Sarda, Lillian Tsai, Jiaming Shen, Yanqi Zhou, Arvind Krishnamurthy, Fan Lai, Henry M. Levy, David E. Culler Abstract . Large language models (LLMs) have excelled in various applications, yet serving them at scale is challenging due to their substantial resource demands and high latency. Our real-world studies reveal that over 70% of user requests to LLMs have semantically similar counterparts, suggesting the potential for knowledge transfer among requests. However, naively caching and reusing past responses leads to a big quality drop. In this paper, we introduce IC-Cache, a caching system that enables live LLM capability augmentation to improve serving efficiency: by leveraging historical request-response pairs from larger models as in-context examples, IC-Cache empowers small LLMs to imitate and even exceed the compositional abilities (e.g., reasoning) of their larger counterparts, enabling selective offloading of requests to reduce cost and latency. Achieving this live augmentation at scale introduces intricate trade-offs between response quality, latency, and system throughput. For a new request, IC-Cache efficiently selects similar, high-utility examples to prepend them to the new request’s input. At scale, it adaptively routes requests across LLMs of varying capabilities, accounting for response quality and serving loads. IC-Cache employs a cost-aware cache replay mechanism that refines example quality offline to maximize online cache utility and efficiency. Evaluations on millions of realistic requests demonstrate that IC-Cache improves LLM serving throughput by 1.4–5.9x and reduces latency by 28–71% without hurting response quality. Conference: ACM SIGOPS 31st Symposium on Operating Systems Principles; October 13–16, 2025; Seoul, Republic of Korea ACM SIGOPS 31st Symposium on Operating Systems Principles (SOSP ’25), October 13–16, 2025, Seoul, Republic of Korea DOI: 10.1145/3731569.3764829 ISBN: 979-8-4007-1870-0/25/10 CCS: Computer systems organization Cloud computing CCS: Computing methodologies Machine learning † † cc-license: by 1. Introduction Large language models (LLMs) have achieved remarkable success in diverse applications like chatbots ( 4 , 3 ) , code generation ( 2 , 10 ) , and math reasoning ( 29 ) , already handling millions of daily user requests ( 68 , 71 ) . While the trend towards ever-larger LLMs enhances service quality—DeepSeek-R1 has 671 billion parameters ( 7 ) —serving these large models is becoming prohibitively costly and operationally complex. At the heart of these challenges lies a fundamental tradeoff between model quality, latency, and serving cost. The drive for high-quality outputs has led to the deployment of models with hundreds of billions of parameters ( 29 , 30 ) . However, their substantial resource demands have led to soaring operational costs for service providers and high latency for users. Ensuring efficient serving is further complicated by the prevalence of bursty workloads, where request loads fluctuate dramatically even in minutes ( 11 , 57 ) . As a result, service providers often have to rely on complex and/or overprovisioned infrastructure to maintain responsiveness ( 7 , 59 ) . Recent advances in LLM serving systems have primarily focused on optimizing system throughput—such as improving parallelism ( 47 , 37 ) , GPU utilization ( 82 , 71 , 18 , 65 ) , and memory efficiency ( 42 , 81 ) —and reducing serving latency (e.g., through better request scheduling ( 63 , 50 , 51 ) ). However, a complementary opportunity remains largely unexplored: exploiting the natural similarity among daily LLM requests . Our analysis of four open-source user request datasets reveals that over 70% of requests have a semantically similar counterpart in past requests. Unfortunately, simply caching historical requests and reusing their responses either yields low hit rates for exact matches or suffers from significant quality degradation in similarity-based matches, as any contextual mismatches can risk off-topic replies (§ 2.3 ). In this paper, we introduce an I n- C ontext Cach ing system, IC-Cache, that unlocks new sweet spots in the quality-latency-cost tradeoff for practical LLM serving as the emergence of long-context small LLMs (1B-10B) ( 67 , 8 , 30 ) . It addresses these classic systems concerns, resource efficiency, load balancing, and response latency, through novel, end-to-end ML-system co-designs. Grounded by in-context learning theory ( 32 , 72 ) , IC-Cache cautiously selects high-utility historical request-response pairs from large LLMs, and prepends them as in-context examples to the new request’s input, guiding smaller LLMs to produce better responses. This enables live capability augmentation of LLMs, where small LLMs can imitate and sometimes even exceed the compositional abilities (e.g., reasoning) of larger models. As a result, low-latency, cost-efficient LLMs can adaptively offload requests from expensive counterparts without compromising response quality. Extending the caching architectures widely adopted in LLM deployments (e.g., prefix caching ( 9 , 6 ) and semantic caching ( 17 ) ), IC-Cache is lightweight, complementary to existing LLM serving systems, and can be integrated with a few lines of code changes (§ 3 ). IC-Cache addresses three key challenges to optimize the quality-latency-cost tradeoff. First, it must efficiently identify high-utility examples to improve model responses for voluminous daily requests ( 68 , 58 ) . Selecting examples based solely on relevance often leads to marginal gains, as it ignores both the model’s intrinsic capabilities and the quality of examples. IC-Cache employs a two-stage example selection mechanism that balances selection efficiency and quality. It first pre-selects a small subset of examples with high relevance to ensure scalability, then uses a lightweight proxy model to estimate their end-to-end utility. While adding more examples can further improve response quality—thus enabling more offloading from large models—it increases the input length, which raises latency and risks exceeding the context window of small models. IC-Cache considers the utility and coverage of examples to optimize offloading efficiency (§ 4.1 ). Second, request offloading to smaller LLMs must account for response quality. Aggressively offloading requests can degrade response quality. Optimizing this efficiency-quality tradeoff requires accounting for request complexity, example utility, and current serving load. Further complicating this is the dynamic nature of real-world deployments, with shifts in request trends (e.g., hot topics), model performance (e.g., after upgrades), and bursty load patterns. IC-Cache introduces a lightweight, bandit-based request router that jointly considers the request and selected examples to route requests to LLMs of varying capabilities. The router adapts to instantaneous load conditions to strike an effective balance between efficiency and quality, and learns from recent requests to refine its policy in a resource- and data-efficient manner (§ 4.2 ). Third, managing the example cache under data and load dynamics requires optimizing example quality and efficiency at scale. The distribution of requests (e.g., new or hot topics) evolves over time, which in turn impacts the utility and access frequency of examples. To maximize overall cache utility, IC-Cache employs a cost-aware example replay mechanism that selectively refines example quality offline—e.g., by opportunistically replaying past examples and preserving those that yield the most effective responses—when the expected gains justify the replay cost. Over time, IC-Cache selectively retains and evicts examples to maintain a bounded cache size while respecting privacy (§ 4.3 ). Our implementation of IC-Cache supports three popular LLM serving frameworks (§ 5 ): HuggingFace Runtime ( 13 ) , vLLM ( 42 ) , and LangChain ( 14 ) . We evaluate IC-Cache across millions of realistic,

Read the complete text