Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism

Vikranth Srivatsa, Zijian He, Pu Guo, Dongming Li, Yiying Zhang · 2026

Read Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism by Vikranth Srivatsa, Zijian He, Pu Guo, Dongming Li, Yiying Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05467.txt.

Introduction

Serving many users from one set of graphics cards means juggling urgent chats against relaxed background jobs. Vikranth Srivatsa and four co-authors present Nitsum, a serving system for large language models. It treats tensor parallelism, the splitting of a model across cards, as a dial it can turn while running.

Nitsum adjusts that split, divides cards between reading and writing stages, and schedules requests together. Weight reuse and fast memory migration make the changes cheap enough to use often. On real workload traces it improved the share of requests meeting their speed targets by up to about five times.

Catalog record

CREATORVikranth Srivatsa, Zijian He, Pu Guo, Dongming Li, Yiying Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.05467

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism Vikranth Srivatsa, Zijian He, Pu Guo, Dongming Li, Yiying Zhang Abstract LLM serving is increasingly multi-tenant: the same deployment must handle latency-critical interactive requests and more relaxed background workloads under a fixed GPU budget. This creates a tiered-SLO setting where maximizing overall goodput (requests that satisfy both TTFT and TPOT targets) is challenging because workload mix, request lengths, and load intensity vary over time. Existing systems mainly optimize request-level controls (e.g., queuing and batching) while keeping execution configuration largely static, which limits adaptation under multi-tier contention. We present Nitsum , a distributed LLM serving system that treats tensor parallelism (TP) as a first-class runtime control surface rather than a static deployment choice. Nitsum jointly optimizes TP level, prefill/decode GPU split, and request scheduling. To make frequent TP adaptation practical, Nitsumintroduces TP-aware weight reuse and fast KV migration. Experiments on real traces and targeted microbenchmarks show that Nitsum improves SLO-compliant goodput over SoTA by up to 5.3 times. 1 Introduction LLM serving is increasingly tiered. A single model deployment [ 9 ] now serves interactive chat [ 13 ] , coding agents [ 7 ] , computer-use agents [ 8 ] , API calls embedded in products [ 6 ] , and long-running background or scheduled jobs [ 11 ] on the same infrastructure. These requests have very different latency expectations: some require fast first-token and steady token rates for user interaction, while others tolerate slower responses in exchange for lower cost [ 39 , 37 , 28 , 20 ] . In practice, this creates tiers of service objectives. Providing tiered-SLO serving would be straightforward if GPU resources were abundant: one could simply provision a dedicated cluster with max capacity for each SLO tier. In practice, however, model providers operate under fixed GPU budgets, while tier mix, request lengths, and load intensity vary substantially over time [ 27 , 44 ] . Strictly separating clusters would therefore waste substantial capacity, while pooling all requests together creates interference among heterogeneous TTFT and TPOT objectives. This leads to the central question of this paper: How should an LLM serving system operate under a fixed GPU budget and maximize the number of requests per second that meet both their TTFT and TPOT SLOs (i.e., goodput) for multiple SLO tiers? Existing SLO-aware LLM serving systems provide part of the answer. Systems such as QLM [ 33 ] , Llumnix [ 40 ] , Chiron [ 32 ] , and SCOOT [ 15 ] support heterogeneous SLOs through request-level control mechanisms, including queuing, batching, migration, and autoscaling. These techniques are effective at prioritizing requests and reducing queueing delay. However, they primarily control when requests run, not how they are executed. The execution configuration of each request, and thus its TTFT and TPOT behavior, remains largely fixed. As a result, these approaches cannot directly change the per-request service time that ultimately determines SLO attainment. We make a key observation: the model execution configuration itself can be used to improve SLO attainment. In particular, Tensor Parallelism (TP) , typically treated as a static knob for fitting models across GPUs, also has a direct impact on SLO-relevant serving performance . Higher TP can reduce prefill latency, improving TTFT for requests with long prompts or tight first-token targets. More surprisingly, we find that higher TP can also improve decode throughput in low-batch-size regimes (but not high-batch-size ones) because of an intra-GPU cache and inter-GPU communication bandwidth tradeoff. This means TP can influence both TTFT and TPOT behavior, making it a practical control surface for LLM serving SLOs rather than merely a deployment choice. Figure 1 : Tiered SLO Workload and Cluster Dynamism. ServeGen [ 44 ] conversation and coding workloads running on 8 H100 GPUs and Llama 3.1-8B. Top row: request pattern and cluster reconfiguration. Middle: optimal cluster configuration of “request group | Prefill/Decode stage | TP level”. Bottom: Nitsum and static TP configuration goodput (higher is better) . Our observation has an important system-level implication. Because TP affects prefill and decode differently, and because different SLO tiers impose different latency pressure, the execution configuration that maximizes SLO-compliant goodput is not fixed. As workload mix and service pressure change over time, the goodput-optimal configuration can shift substantially. Figure 1 illustrates this effect: under a realistic workload from Alibaba cloud [ 44 ] , both request demand and optimal cluster configuration vary continuously, and no single static TP setting achieves the best performance. Clearly, systems that can adapt their execution configuration at runtime can achieve significantly higher goodput. However, doing so is challenging in practice. Naively changing TP requires weight reloading, kernel reinitialization, and KV-cache movement across GPUs, which can take seconds to tens of seconds. These overheads are large relative to the duration of typical workload bursts, as shown in Figure 1 , and they can negate any potential benefit. In fact, from our experiments, today’s system ends up spending most of its time reconfiguring rather than serving requests, effectively driving goodput close to zero. Thus, the central challenge is to make execution-level TP adaptation fast enough to track workload dynamics. In this paper, we present Nitsum , a distributed LLM serving system that addresses this problem. Nitsum maximizes overall system goodput under a fixed GPU budget by jointly reconfiguring TP level, prefill/decode GPU allocation, and request placement at runtime, while maintaining fairness across SLO tiers. Notably, our cluster adaptation and scheduling policy treats TP reconfiguration as effectively cost-free, which greatly simplifies the optimization. Realizing this abstraction in practice requires reducing today’s TP switching overhead from seconds or even tens of seconds to near zero. Nitsum introduces two key enabling mechanisms to achieve near-zero TP switching overhead. First, it eliminates model reloads during TP changes by storing one full weight copy per GPU and selecting TP-specific shards at execution time, while keeping TP-specialized execution processes warm. Second, it bounds TP-transition latency with an aggregated, pipelined KV migration mechanism that makes stop-and-migrate reconfiguration fast enough for sub-second-level control. With switching overhead reduced to this level, Nitsum can treat TP as an “almost-free” runtime knob rather than a static deployment choice. On top of these mechanisms, Nitsum performs goodput-aware cluster reconfiguration and request scheduling. It uses offline prefill/decode profiles together with recent arrival statistics and tier-specific TTFT/TPOT SLOs to estimate the SLO-compliant throughput of candidate configurations. It then assigns GPUs across tiers and stages using a weighted greedy policy that balances goodput efficiency with fairness across tiers. The resulting configuration is enforced by global and local schedulers that coordinate request feasibility, placement, and iteration-level batch formation. We evaluate Nitsum against four baselines: SGLang [ 19 ] , Llumnix [ 40 ] , Chiron [ 32 ] , and a separated per-SLO-tier cluster setup. Our results on two GPU platforms, two LLMs, and two real traces [ 27 , 44 ] show that Nitsum consistently improves goodput over all baselines, by up to 5.3 \times . In regimes where prior baselines collapse to near-zero goodput, Nitsum continues to sustain substantial SLO-compliant throughput. In summary, this paper makes the following contributions: • Detailed analysis of how TP affects TTFT and TPOT for different request/batch sizes. • The proposal of using TP as an effective

Read the complete text