Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads

Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, and Samyam Rajbhandari Snowflake AI Research · 2025

Read Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads by Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, and Samyam Rajbhandari Snowflake AI Research (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16495.txt.

Introduction

Serving language models faces a tug of war: splitting each model across chips answers fast but wastes throughput, while copying it whole answers slowly at peak load. Mert Hidayetoglu and five co-authors notice a training trick whose memory layout stays fixed, which lets it mix with the splitting approach. Their Shift Parallelism switches between the two modes as traffic changes.

In quiet periods it favors fast replies; in busy ones it favors volume. On real production traces it answered interactive work about half again as fast and raised batch throughput by half. Tests across models and loads tell the same story.

Catalog record

CREATORMert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, and Samyam Rajbhandari Snowflake AI Research
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-16495

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Shift Parallelism: Low-Latency, High-Throughput LLM Inference for Dynamic Workloads Mert Hidayetoglu, Aurick Qiao, Michael Wyatt, Jeff Rasley, Yuxiong He, and Samyam Rajbhandari Snowflake AI Research Abstract . Efficient parallelism is necessary for achieving low-latency, high-throughput inference with large language models (LLMs). Tensor parallelism (TP) is the state-of-the-art method for reducing LLM response latency, however GPU communications reduces combined token throughput. On the other hand, data parallelism (DP) obtains a higher throughput yet is slow in response latency. Best of both worlds does not exist, and it is not possible to combine TP and DP because of the KV cache variance across the parallelisms. We notice Sequence Parallelism (SP—Ulysses in training) has similar properties as DP but with KV cache invariance. We adapt SP to inference, and combine it with TP to get the best of both worlds. Our solution: Shift Parallelism. Shift Parallelism dynamically switches across TP and SP, and minimizes latency in low traffic without losing throughput in high traffic. The efficient GPU communications of Shift Parallelism yields up to i) 1.51 × \times faster response in interactive workloads and ii) 50% higher throughput in batch workloads, compared to a TP-only solution. We evaluate Shift Parallelism with real-world production traces with dynamic traffic patterns as well as synthetic benchmarking patterns across models, context sizes, and arrival rates. All results affirm the same: Shift Parallelism has a better the latency vs. throughput tradeoff than TP or DP, and hence obtains low latency without degrading throughput in dynamic workloads. 1. Introduction LLM inference has become the dominant workload in AI as its applications span agentic systems, chatbox (interactive) applications, model post-training (e.g. reinforcement learning), and image/video generation. The efficiency of inference systems is critical to both the performance and cost of AI applications. As of today, GPU parallelization is the prominent way of enabling large-scale AI production, and advanced multi-GPU parallelization techniques make nontrivial tradeoffs across key performance metrics. The parallelism techniques for inference are largely inherited from training, yet inference is different from training in terms of workload characteristics. Training workloads are typically homogeneous, stable, and do not care about latency but only care about throughput. Inference, on the contrary, is bursty and dynamic, and often has unpredictable traffic patterns. Furthermore, different workloads have different performance requirements. As a result, leveraging parallelism techniques designed for training in inference results in complex performance and cost trade-offs. Inference Workload Characteristics When people talk about inference systems, they often refer to interactive workloads, or batch workloads. Interactive workloads process requests with low concurrency to minimize the completion time of each request. The completion time latency is important when there is a chain of interactions between the user and the LLM, such as in REST applications. The completion latency depends on the time to first token (TTFT)—and time per output token (TPOT), which are both critical in real-time applications. Batch workloads involve a number of requests to be processed concurrently, and the latency of an individual request is not critical. For example, workloads such as batched summarization or translation of hundreds or thousands of documents can cause high-traffic bursts that require high combined throughput of input and output tokens to minimize the cost per token. In enterprise systems, the request traffic pattern is often mixed and dynamically changes over time in unpredictable ways. In such dynamic settings, it is a challenge to optimize for different traffic patterns simultaneously, since existing parallelization techniques impose significant trade-offs. Figure 1 . Comparison of response speed (#input tok./TTFT) and generation rate (1/TPOT), and throughput (tokens/sec). Shift Parallelism obtains a higher throughput than TP in high traffic, and lower latency than TP and DP in low traffic. Latency vs. Throughput vs. Cost Tradeoff Existing parallelisms exhibit prohibitive latency vs. throughput (cost) trade-offs, as explained below. Tensor parallelism (TP) partitions the model weights and computation in each layer. It has to synchronize the embeddings across layers with costly all-reduce communications. By splitting model weights and computation across GPUs, it optimizes for latency (i.e., TTFT and TPOT), yet the communication overhead increases the cost (i.e., reduces throughput). Data parallelism (DP) parallelizes across request boundaries in embarrassingly parallel, providing high throughput. Yet, DP cannot speed up work within a single request, and therefore unsuitable for highly interactive workloads. The first two rows of Table 1 shows the performance tradeoffs of TP and DP. We do have the choice to deploy TP and DP in separate nodes and route latency- and throughput-oriented requests, respectively. However, duplicating the node count (one for TP and one for DP) doubles the deployment cost and add complexity. Why can’t we combine both? A performant and low-cost inference system should be able to switch between latency-oriented and throughput-oriented parallelisms swiftly in a single deployment based on traffic demands. But this is not viable with TP and DP because they have different attention layouts. Specifically, their KV cache memory layouts are incompatible, and switching requires complex and costly data movement. However, in this work, we notice that Sequence Parallelism (SP) ( Jacobs et al. 2023 ) —another form of parallelism developed and used in training—can offer a potential solution to resolving the challenge. SP splits the input sequence across GPUs to parallelize work within a single request to reduce TTFT. Unlike TP, it avoids costly all-reduce communication, while still achieving high GPU utilization. And while SP cannot parallelize decoding steps, resulting in the worst TPOT compared to TP and DP (Table 1 ), it has the same KV cache layout as TP, allowing for dynamic switching to TP when TPOT is critical. We call this dynamic approach Shift Parallelism, which is the focus of this work. Table 1 . Performance tradeoffs of inference parallelisms. Shift Parallelism dynamically chooses the parallelization strategy between SP and TP based on the real-world traffic pattern, identified by the number of batched tokens in each iteration. By a given threshold, Shift Parallelism uses: • TP for small batches—minimizing TPOT. • SP for large batches—minimizing TTFT and achieving near-optimal throughput. This is possible because the KV cache memory layout remains invariant between TP and SP, allowing Shift Parallelism to switch modes seamlessly, based on batch size and traffic patterns. More specifically, the KV cache layout does not change when switching across SP and TP. Figure 1 benchmarks the latency and throughput tradeoffs of related parallelisms. Shift parallelism provides 1.5 \times higher throughput than TP in high traffic and 1.5 \times faster response in low traffic, 2 \times faster generation than DP in low traffic while losing only 17% throughput in high traffic. In this paper, we (1) We characterize inference workloads and identify latency vs. throughput tradeoffs with existing inference parallelization (TP, DP). (2) Adapt SP from training to inference and generalize it for a diverse set of inference models supporting GQA ( Ainslie et al. 2023 ) , load-balancing at small batch sizes, combination of TP, and KV cache replication when paralleism degree is higher than number of KV heads. (3) Propose Shift Parallelism for dynamically switching across SP and TP for mitigating latency vs througput tradeoff for dynamic workloads. (4) Test Shift Parallelism with real-world production workloads and evaluate the performance characteristics via extensive benchmarking demonstrating up to 1.5 \times faster response time

Read the complete text