Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles

Moiz Arif 1 , Avinash Maurya 2 , Sudharshan Vazhkudai 1 , Bogdan Nicolae 2 · 2026

Read Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles by Moiz Arif 1 , Avinash Maurya 2 , Sudharshan Vazhkudai 1 , Bogdan Nicolae 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19775.txt.

Introduction

Arif, Maurya, Vazhkudai and Nicolae map why inference for reasoning models, which emit long chains of thought, strains systems differently from ordinary generation. Testing models from eight billion to six hundred seventy-one billion parameters, they find data parallelism throttles early as key-value cache fragments, while tensor parallelism frees stranded memory near the thirty-two-billion crossover. Dense giants stay bound by links and memory bandwidth; sparse mixture-of-experts models instead suffer routing delays. The study offers a decision guide for reasoning-era infrastructure.

Catalog record

CREATORMoiz Arif 1 , Avinash Maurya 2 , Sudharshan Vazhkudai 1 , Bogdan Nicolae 2
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58405
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19775

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles Moiz Arif 1 , Avinash Maurya 2 , Sudharshan Vazhkudai 1 , Bogdan Nicolae 2 Abstract The transition from standard generative AI to reasoning-centric architectures , exemplified by models capable of extensive Chain-of-Thought (CoT) processing, marks a fundamental paradigm shift in system requirements. Unlike traditional workloads dominated by compute-bound prefill, reasoning workloads generate long chains of reasoning tokens that shift inference into a Capacity-Bound regime . This paper presents a comprehensive system characterization, evaluating models ranging from 8B to 671B parameters on GPUs clusters. By systematically exploring the interplay between Data, Tensor, and Pipeline parallelism, we identify critical bottlenecks that defy standard scaling heuristics. Our analysis reveals that data parallelism is throughput efficient for small models but hits a capacity trap on reasoning workloads as KV-cache fragmentation forces early throttling resulting in sub-optimal compute utilization. Tensor parallelism unlocks stranded memory and delivers sublinear gains near the 32B crossover. At frontier scale, dense models (e.g., Llama-405B) are interconnect and memory-bandwidth bound and favor high-degree TP, while sparse Mixture-of-Experts (MoE) models (e.g., DeepSeek-R1) are limited by routing and synchronization latency and benefit from hybrid strategies. These insights provide a rigorous decision framework for navigating the reasoning cliff, establishing new architectural imperatives for the next generation of inference infrastructure. I Introduction I-A Motivation The deployment of Large Language Models (LLMs) has transitioned from a capability demonstration to a cornerstone of industrial computing. However, the emergence of reasoning-centric architectures , exemplified by DeepSeek-R1 [ 16 ] and OpenAI’s o1 [ 28 ] marks a fundamental paradigm shift in system requirements. Unlike standard generative models that prioritize fluent text generation, reasoning models employ CoT [ 35 ] like processing to generate extensive intermediate logic traces before producing a final answer. This shift fundamentally alters the resource signature of AI inference. While traditional LLM serving is often dominated by compute-bound prefill or bandwidth-bound decoding of short responses, reasoning workloads typically demonstrate a Capacity-Bound inference regime. The generation of thousands of intermediate “thinking” tokens [ 34 ] creates massive, persistent Key-Value (KV) cache footprints that saturate High Bandwidth Memory (HBM) long before compute utilization peaks [ 37 ] . Consequently, standard scaling heuristics such as maximizing batch size for throughput or relying solely on Data Parallelism (DP) become counter-productive, triggering severe memory thrashing, scheduler preemption, and non-linear latency spikes [ 1 , 23 ] . Deploying these frontier-scale models (often exceeding 400B parameters [ 18 , 25 ] ) requires navigating a complex, high-dimensional design space [ 21 ] . System architects must balance the low-latency requirements of interactive reasoning against the strict capacity limits of modern accelerators. This necessitates a move beyond monolithic scaling toward nuanced combinations of Data, Tensor [ 33 ] , and Pipeline parallelism [ 20 ] , tailored specifically to the sparsity and sequence length characteristics of the model [ 39 ] . I-B System Challenges in Reasoning Inference Deploying reasoning-centric models at scale introduces distinct system-level bottlenecks that are structurally different from training [ 6 ] or standard chat inference: • The Capacity Wall: Reasoning traces with Output Sequence Length ( OSL)\gg 10k cause the KV cache to grow linearly, rapidly exhausting per-GPU HBM [ 24 ] . In DP with engines such as vLLM [ 22 ] , this leads to “stranded capacity” where fragmented memory across replicas forces premature request throttling despite low compute utilization [ 37 ] . • The Parallelism Efficiency Gap: While Tensor Parallelism (TP) alleviates memory pressure by using collective HBM of multiple GPUs, it incurs communication overheads that degrade performance beyond intra-node scaling [ 10 ] . Conversely, Pipeline Parallelism (PP) offers memory relief but suffers from “pipeline bubbles” that are difficult to hide in serial, auto-regressive generation tasks [ 3 ] . • Architectural Divergence: The optimal strategy is no longer universal as we show in our analysis. As we demonstrate, dense models (e.g., Llama-3.1-405B) behave fundamentally differently from sparse MoE models (e.g., DeepSeek-R1-671B), requiring bespoke parallelization hierarchies to align with their respective compute-to-communication ratios [ 8 ] . While recent work has improved kernel efficiency and KV management, the dominant scaling limits for reasoning workloads increasingly arise from system-level capacity and scheduling dynamics rather than operator inefficiencies. This work isolates and quantifies these limits under realistic inference conditions. I-C Contributions This paper presents a systematic characterization of GPU-based inference for reasoning-centric LLMs. We evaluate models ranging from 8B to 671B parameters on a cluster of NVIDIA H200 GPUs, identifying the breakpoints where standard scaling laws fall short. Our specific contributions are: 1. Characterization of the “Parallelism Transition Point”: We empirically determine the model size and sequence length thresholds where DP collapses due to KV saturation, necessitating a transition to hybrid or TP approaches. 2. Quantification of the “Reasoning Gap”: We demonstrate how the extended decoding phases of reasoning models shift the critical bottleneck from prefill compute (TTFT) to decode memory capacity (TPOT). 3. Evaluation of Frontier Architectures: We provide a comparative analysis of large dense versus sparse scaling, revealing that MoE architectures favor hybrid strategies (PP+TP) to mitigate synchronization costs, whereas dense models demand high-bandwidth TP configurations. 4. Operational Guidance: Based on these insights, we provide guidance for selecting optimal parallelism strategies that maximize fleet-level throughput while meeting strict latency SLAs for reasoning tasks. II Background and Related Work II-A Reasoning Inference Breakdown Reasoning-centric inference is distinct from standard chat workloads in its execution flow and resource demands. It proceeds in two phases with orthogonal hardware requirements: Prefill (The Compute Phase) The model processes the user prompt (input sequence) in parallel and dominated by large matrix-matrix multiplications (GEMMs) and is strictly compute-bound. Modern GPUs effectively utilize their tensor cores, achieving high Streaming Multiprocessor (SM) occupancy. The latency of this phase determines the TTFT. Decode (The Memory Phase) Following prefill, the model enters the autoregressive generation loop. Unlike standard chat ( OSL\approx 500 ), reasoning traces can exceed 10000 tokens [ 36 ] . Each token generation requires reading the entire model weight set and the active KV cache from HBM [ 37 ] . This phase is strictly bandwidth-bound ; the arithmetic intensity collapses, and the GPU spends the majority of cycles performing memory reads. Our telemetry indicates that for reasoning workloads, the system spends >99\% of its wall-clock time in this inefficient regime [ 2 ] . II-B Model Architectures and Attention Mechanisms The impact of the GPU memory exhaustion is modulated by the specific architecture of the model. We characterize two distinct paradigms found in state-of-the-art reasoning engines: Dense Architectures (Grouped-Query Attention) Models like Llama-3.1-405B [ 18 ] utilize a dense transformer architecture where every parameter is active for every token. To mitigate memory pressure, these models employ Grouped-Query Attention (GQA) (typically 8 KV heads) [ 4 ] , which reduces the KV footprint by 3\times – 8\times compared to standard Multi-Head Attention (MHA) [ 12 ] . However, the memory cost remains linear with layer count [ 5 ] . For the 405B model ( \approx 126 layers), the KV cache consumes \approx 1.05 MB per token in FP16. Serving a batch of 128 requests with 10k reasoning tokens each consumes over 1.3 TB of memory solely for the

Read the complete text