Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning
Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu *, Junyi Shu *, Jiarong Xing *, Ying Sheng *, UCLA · 2025
Read Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning by Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu *, Junyi Shu *, Jiarong Xing *, Ying Sheng *, UCLA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04021.txt.
Introduction
Serving many language models at once means most of them sit idle most of the time, and Shan Yu and co-authors attack the wasted memory. Production traces show demand arrives in bursty groups: clusters of models flare up together, then go quiet. Prism, their serving framework, treats graphics memory as elastic, ballooning it away from idle models and toward busy ones, so one scheme covers both sharing space and sharing time. Its driver, kvcached, is open source and already runs in production on more than ten thousand graphics processors.
The authors argue memory, not compute, is the binding constraint in multi-model serving as token prices fall. Reclaiming it on the fly keeps rare but essential models available without paying to keep them resident.
Catalog record
| CREATOR | Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu *, Junyi Shu *, Jiarong Xing *, Ying Sheng *, UCLA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-04021
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning Shan Yu, Yifan Qiao, Mingyuan Ma, Yangmin Li, Shuo Yang, Xinyuan Tong, Yang Wang, Zhiqiang Xie, Yuwei An, Shiyi Cao, Ke Bao, Deepak Vij, Xiaoning Ding, Yichen Wang, Qingda Lu, Zhong Wang, Gao Gao, Harry Xu *, Junyi Shu *, Jiarong Xing *, Ying Sheng *, UCLA Abstract Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall. Analysis of production traces reveals a dynamic bursty-group pattern in which sets of models become active together and shift over time; existing space- and time-sharing approaches lack principled mechanisms to adapt to this variability, forcing trade-offs between SLO adherence and efficiency. We observe that elastic memory allocation can unify spatial and temporal sharing. Based on this insight, we have developed Prism, a memory-centric LLM co-serving framework that applies memory ballooning to reclaim memory across models and support both forms of sharing under a single scheme. Prism’s balloon driver, referred to as kvcached , has been open-sourced at https://github.com/ovg-project/kvcached , and deployed in production environments across 10K+ GPUs. † † * Corresponding authors. 1 Introduction Serving LLMs is costly for providers such as Hugging Face and Alibaba Cloud, which must host thousands of base and fine-tuned models, many with low request volumes yet mandatory availability [ 52 , 42 , 53 ] . As token prices fall and models vary widely in size and workload patterns, reducing inference cost while preserving performance has become a central objective. The core challenge, however, is not hardware expense but chronic under-utilization: to satisfy strict SLOs, industry practice often dedicates a model-parallel GPU group ( i.e. , one or more GPUs jointly serving a single model instance), guaranteeing immediate responsiveness but wasting substantial capacity. This strategy is particularly inefficient for models with bursty or sparse traffic, and production analyses show that GPU duty cycles commonly fall below 30% [ 52 , 53 ] . To improve utilization, prior work has explored various GPU sharing mechanisms at different granularities. Because auto-regressive LLM inference is intrinsically memory-bound, tail latency is governed primarily by memory stalls ( e.g. , page faults, PCIe migrations). Moreover, compute sharing is already well studied—both via hardware mechanisms such as NVIDIA MIG [ 29 ] and vGPU [ 30 ] , and via software solutions [ 27 , 61 , 13 , 10 , 12 , 41 ] . In contrast, effective memory sharing remains comparatively underexplored. We therefore center this work on advancing effective GPU memory sharing. The work of memory sharing for LLM serving falls into two categories: space sharing and time sharing. Space sharing ( e.g. , MuxServe [ 11 ] ) colocates multiple models on the same GPU to utilize unused memory and compute cycles, which is effective for smaller, low-traffic models that are continuously accessed. Time sharing ( e.g. , QLM [ 33 ] and Aegaeon [ 52 ] ) swaps models in and out of GPU memory, which is ideal for handling sporadic requests to models as the resources can be freed up when models are idle. While these strategies are effective for specific cases, our analysis of a broad range of production serving traces demonstrates that they fail to adapt to more complex and realistic usage scenarios. In particular, we have conducted a comprehensive analysis of production traces drawn from major inference providers ( i.e. , Hyperbolic and Novita AI) as well as independent evaluation platforms ( i.e. , Chatbot Arena). These traces encompass 11 days to 16 months of traffic across 58 heterogeneous models. Our results indicate that real-world workloads display diverse, rapidly changing patterns that elude any single, fixed sharing mechanism. Key observations. At the model level , deployed models show a bursty-group behavior: when a burst occurs, requests concentrate on a particular subset of models, and that subset shifts over time, similar to an application’s dynamically changing working set. This pattern is driven by modern compound AI systems and agentic pipelines, which depend on a central reasoning LLM together with a number of fine-tuned or LoRA models for specialized agent tasks. At the request level , detailed inspection of the traces confirms that requests are extremely dynamic and volatile and fluctuate rapidly and unpredictably, limiting the value of existing GPU-sharing techniques. Space sharing, alone, fails to adapt to longer idle periods; for instance, in the Novita AI trace, where models are idle for over 70% of the time, space sharing creates significant memory fragmentation, leaving 50% of GPU memory occupied but unused. In contrast, time sharing, alone, struggles with rapid workload fluctuations. The significant latency overhead of swapping ( e.g. , several seconds) causes severe thrashing during traffic bursts, resulting in a spike of requests violating their latency SLOs due to queuing delays. Maximizing resource utilization while preserving SLO attainment requires integrating both strategies: space sharing for colocating low-traffic models and time sharing for burst handling and idle-resource reclamation. However, mingling these approaches is far from trivial due to the fundamentally different challenges they must address—the flexibility of memory balancing for space sharing vs. the efficiency of model swapping for time sharing. One could attempt to switch spatial and temporal sharing strategies dynamically; yet, co-locating models with conflicting sharing needs creates resource contention. Specifically, a model requiring time-sharing typically relies on preempting resources to maximize burst performance, which directly contradicts the persistent resource availability required by a model colocated through space-sharing. Without a mechanism to fluidly mediate these conflicting operational modes, the system is forced into rigid configurations that cannot adapt to dynamic load changes. Insight: a memory-centric view to unify space- and time-sharing. Our central observation is that GPU memory is the unifying bottleneck, and each sharing strategy is concerned with a different portion of that memory : time-sharing focuses on efficiently swapping in LLM weights , whereas space-sharing determines how to scale KV-cache capacity among concurrently served models. This perspective parallels memory ballooning in virtualization, where a hypervisor dynamically reclaims memory from guest VMs and reallocates it where needed. In the context of LLM co-serving, an analogous “balloon driver” can reclaim memory from idle models to swap in the weights of new models (supporting time-sharing) or from models with low request rates by shrinking their KV-cache reservations (supporting space-sharing). Treating GPU memory as an elastic resource allows the system to fluidly shift between sharing modes and even enable both simultaneously ( i.e. , apply time-sharing to only a subset of colocated models), ensuring active models obtain the memory required to satisfy their SLOs while maximizing cluster-level efficiency. Prism. We developed Prism, a new memory-centric GPU sharing system for multi-LLM serving. At its core, Prism implements the concept of memory ballooning for LLMs to enable demand-aware, cross-model memory sharing, a capability missing in existing systems but critical for unifying spatial and temporal strategies. Prism ensures that models can promptly expand their memory footprint to meet latency SLOs, while unused memory is efficiently harvested from others to maximize overall cluster utilization. The design of Prism addresses two key challenges. Challenge 1: How to enable flexible memory ballooning for LLMs? Today’s LLM serving engines ( e.g. , SGLang and vLLM) are designed for single-model serving and