ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management
Zaifeng Pan, Yipeng Shen, Zhengding Hu, Zhuang Wang, Aninda Manocha, Zheng Wang, Zhongkai Yu, Yue Guan, Yufei Ding · 2026
Read ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management by Zaifeng Pan, Yipeng Shen, Zhengding Hu, Zhuang Wang, Aninda Manocha, Zheng Wang, Zhongkai Yu, Yue Guan, Yufei Ding (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21473.txt.
Introduction
Zaifeng Pan and eight co-authors make large multi-agent simulations cheaper to run. Simulations where each agent is a language model are spreading across many fields. Each agent keeps private state on the graphics chip, and memory fills fast as the agent count grows.
The authors notice two helpful traits of these workloads. Only a few agents are active at any moment, and the order of future calls can be estimated. Their system, ScaleSim, manages memory based on how soon each agent will be called next. That foresight keeps the chip from choking as simulations scale up.
Catalog record
| CREATOR | Zaifeng Pan, Yipeng Shen, Zhengding Hu, Zhuang Wang, Aninda Manocha, Zheng Wang, Zhongkai Yu, Yue Guan, Yufei Ding |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21473
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ScaleSim: Serving Large-Scale Multi-Agent Simulation with Invocation Distance-Based Memory Management Zaifeng Pan, Yipeng Shen, Zhengding Hu, Zhuang Wang, Aninda Manocha, Zheng Wang, Zhongkai Yu, Yue Guan, Yufei Ding Abstract LLM-based multi-agent simulations are increasingly adopted across application domains, but remain difficult to scale due to GPU memory pressure. Each agent maintains private GPU-resident states, including models, prefix caches, and adapters, which quickly exhaust device memory as the agent count grows. We identify two key properties of these workloads: sparse agent activation and estimable agent invocation orders. Based on an analysis of representative workload classes, we introduce invocation distance, a unified abstraction that estimates the relative order in which agents will issue future LLM requests. Leveraging this abstraction, we present ScaleSim, a memory-efficient LLM serving system for large-scale multi-agent simulations. ScaleSim enables proactive prefetching and priority-based eviction, supports diverse agent-specific memory through a modular interface, and achieves up to 1.74 × \times speedup over SGLang on simulation benchmarks. † † affiliationnotice: Equal contribution 1 Introduction Large language model (LLM)-based multi-agent simulation is widely used to model and analyze complex systems involving interacting agents. It has enabled progress in diverse application domains, including social science ( 35 ; 36 ; 39 ; 21 ) , autonomous driving ( 11 ; 55 ) , urban planning ( 27 ) , robotics ( 15 ) , and economic modeling ( 18 ) . By capturing how individual agents perceive, decide, and act in shared environments, multi-agent simulation serves as a key tool for studying collective behaviors, coordination strategies, and emergent dynamics in large-scale systems. Scaling up the number of agents in simulation improves behavioral diversity and enables more complex interaction patterns, which are critical for many downstream tasks. However, existing backend systems for serving LLM-based agent applications are struggling to handle large-scale simulations efficiently. A key bottleneck arises from the agent-specific memory , which includes private models ( 34 ) , LoRA adapters ( 49 ) , prefix caches ( 52 ; 9 ; 33 ) , and action histories 1 1 1 To avoid ambiguity, we use action history to represent the agent’s accumulated experience or memory content, as memory in this paper refers to storage resources such as GPU memory. ( 30 ; 5 ; 46 ) . As illustrated in Figure 1 , each agent maintains its own memory components on the GPU. As the agent population increases, the aggregate memory demand grows rapidly, often exceeding available GPU capacity. This results in frequent memory evictions and data transfers between host and device, introducing significant I/O overhead (Figure 3 ) and degrading overall system throughput. Figure 1 : Illustration of multi-agent simulation phases and agent-specific memory consumption. Despite this growing memory footprint, we make a key observation: agent execution is inherently sparse. Specifically, the number of concurrent LLM requests at any given time is much smaller than the total number of agents. This sparsity arises from the two-phase structure of agent simulation shown in Figure 1 : one phase involves invoking the LLM to generate actions (LLM-involved), while the other executes those actions in the environment without requiring LLM inference (LLM-not-involved). We refer to an agent as activated when it is in the LLM-involved phase. Our profiling of large-scale agent societies ( 37 ) reveals that only a small fraction of agents are activated at any moment, resulting in sparse LLM execution patterns shown in Figure 3 . This execution sparsity implies that simply adding more machines to handle scalability often leads to poor hardware utilization, since most agents are inactive at any given time. Figure 2 : I/O overhead per simulation step increases with the number of simulation agents. Figure 3 : Only a small fraction of agents are activated during each simulation step. We make another key observation: in many multi-agent simulation workloads, the relative order in which agents are invoked can be effectively estimated. Based on how agent invocation order can be estimated, we categorize representative applications into three types: (1) Independent simulation , where agents operate in isolation and the next invocation can be inferred directly from their current action; (2) Interaction-involved simulation , where agent interactions are possible, but future activations remain predictable based on environmental cues such as agent location and moving velocity; and (3) Predefined activation paths , such as information diffusion in social networks, where the next activation time of agents can be compared through structural patterns like hop counts. Motivated by sparse agent activation and estimable invocation orders, we propose ScaleSim, an LLM serving system that efficiently manages GPU memory for large-scale multi-agent simulations. ScaleSim introduces invocation distance for each agent to guide memory management and exposes a unified interface for applications to provide distance estimates. Based on the invocation distances of agents, ScaleSim enables distance-aware eviction, proactive prefetching, and execution preemption, which substantially reduces I/O overhead. ScaleSim further defines an extensible abstraction for agent-specific memory, allowing heterogeneous memory modules to be seamlessly integrated and jointly managed. We implement ScaleSim on top of SGLang and evaluate it on three representative simulation applications: AgentSociety ( 37 ) , Generative Agents ( 35 ) , and information diffusion ( 8 ) . Experimental results demonstrate that ScaleSim provides consistent performance improvements as the number of agents scales. In summary, we make the following contributions: • We identify two key properties of multi-agent simulation workloads: sparse agent activation and estimable agent invocation orders. To capture these behaviors, we introduce the unified invocation distance abstraction, which generalizes activation patterns across diverse applications. • We design and implement ScaleSim, a memory-efficient LLM serving system that leverages invocation distance to guide memory management decisions. • We conduct comprehensive experiments on three representative workloads. Results show that ScaleSim achieves up to 1.74 \times speedup compared to a vanilla SGLang baseline, demonstrating its effectiveness in scaling multi-agent simulation. 2 Background and Related Work LLM-based multi-agent simulation. LLM-based multi-agent simulation models a system composed of multiple autonomous agents, where each agent relies on a large language model to reason about its local state and generate actions in response to the shared environment. As illustrated in Figure 1 , the behavior of each LLM-based agent typically alternates between two phases: invoking the LLM to generate the next action, and executing that action over a period of time without LLM involvement. This results in the sparse activation patterns shown in Figure 3 , where only a small subset of agents are simultaneously issuing LLM requests. This structural property introduces opportunities for more efficient memory and compute management in large-scale systems. Agent-specific memory. Each agent in an LLM-based simulation maintains its own memory footprint, comprising both static model configurations and dynamic runtime state. On the static side, agents may load personalized components such as LoRA adapters ( 49 ) , system-specific prefix caches that encode role and behavioral priors ( 52 ; 31 ; 9 ; 32 ; 33 ) , or even auxiliary draft models used for speculative decoding ( 17 ; 24 ; 26 ) . These components are typically instantiated per agent and persist across multiple simulation steps. At runtime, agents accumulate action histories based on prior interactions ( 30 ; 29 ) , often requiring dedicated storage and indexing structures (