From Attention to Disaggregation: Tracing the Evolution of LLM Inference
Srinivasa Rao Aravilli, Mustafa Saify, Madabattula Rajesh Kumar, Shashank Srivastava · 2025
Read From Attention to Disaggregation: Tracing the Evolution of LLM Inference by Srinivasa Rao Aravilli, Mustafa Saify, Madabattula Rajesh Kumar, Shashank Srivastava (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07422.txt.
Introduction
Running huge language models now costs more effort than training them, argue Madabattula Rajesh Kumar and colleagues. Their survey traces serving systems from the Transformer breakthrough to models with trillions of parameters. Memory bandwidth, speed, and delay limits now dominate design choices.
The authors explain why splitting inference into separate stages helps. The compute-heavy prefill stage, which reads the prompt, is split from the memory-heavy decode stage, which writes the reply. Each part can then scale on its own hardware. Framed as balancing delay, throughput, and cost, disaggregation becomes a classic distributed systems problem with practical payoffs.
Catalog record
| CREATOR | Srinivasa Rao Aravilli, Mustafa Saify, Madabattula Rajesh Kumar, Shashank Srivastava |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07422
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Attention to Disaggregation: Tracing the Evolution of LLM Inference Srinivasa Rao Aravilli, Mustafa Saify, Madabattula Rajesh Kumar, Shashank Srivastava Abstract . The 2017 introduction of the Transformer architecture through the paper "Attention Is All You Need" fundamentally transformed sequence modelling by replacing recurrent neural networks with parallelizable self-attention mechanisms, enabling unprecedented scalability. Models such as GPT-1 and BERT showcased the power of large-scale pre-training, while GPT-3, with 175 billion parameters, revealed emergent few-shot and zero-shot learning capabilities. However, as model sizes grew into the hundreds of billions and trillions of parameters, the inference process real-time output generation became the primary bottleneck, constrained by computational throughput, memory bandwidth, and low latency requirements for interactive use cases. Deploying Large Language Models (LLMs) goes far beyond software or model design, it is a complex distributed systems challenge that requires seamless integration of specialized hardware, efficient network scheduling, intelligent resource management, and automated scaling. Fundamentally, LLM inference constitutes a constrained multi-objective optimization problem where latency (e.g., time to first token and total latency) must be minimized, throughput (requests per second) maximized, and cost (dollars per token) reduced. All while adhering to strict service-level agreements and operating within the limits of heterogeneous GPU clusters. This architectural shift, known as "disaggregated inference," is as revolutionary for AI as microservices were for web applications. By decomposing formerly monolithic tasks into modular components that can operate across thousands of GPUs or specialized accelerators, it unlocks significant gains in efficiency, reliability, and agility. In contrast, traditional monolithic inference pipelines relying on homogeneous GPU hardware encounter bottlenecks caused by competing resource demands between the compute-intensive "prefill" phase and the memory-intensive "decode" phase. These approaches also suffer from inflexible scaling, single points of failure, and energy inefficiencies when handling diverse and dynamic workloads. This paper explores the evolution toward a disaggregated inference architecture that incorporates distributed systems principles, including service decomposition, resource disaggregation, workload partitioning, and orchestration across heterogeneous hardware. By decoupling prefill and decode phases into independently scalable clusters, this approach mitigates resource contention and enables independent optimization of key performance metrics like Time-to-First-Token (TTFT) and Inter-Token Latency (ITL). We first analyse six key inference optimizations—Key-Value Caching, Flash Attention, Continuous Batching, Speculative Decoding, Paged Attention, and Radix Attention outlining their mechanisms, historical adoption, and trade-offs in terms of the CAP theorem (Consistency, Availability, Partition Tolerance). We illustrate how these optimizations align with distributed systems concepts such as caching, parallelism, and eventual consistency, while highlighting their constraints in asynchronous environments. Ultimately, we advocate for a disaggregated paradigm that leverages compute and memory-optimized clusters, dynamic routing, and fault isolation to achieve elastic scalability, reduced latency, and enhanced robustness. This is supported by an in-depth technical survey of three leading frameworks, each representing a distinct archetype: DistServe, a research-first system focused on "goodput" optimization. AIBrix, a cloud-native production orchestration framework and NVIDIA Dynamo, a modular platform for enterprise-scale deployments. Our analysis examines their architectural innovations, implementation strategies, and comparative advantages to guide deployment decisions. Figure 1. Timeline of LLM Inference Optimization: From Quantum Leaps to Disaggregated Efficiency 1. Introduction Figure 2. A quantum leap in AI 1.1. Evolution from Transformers to Large Language Models: A quantum leap in AI In 2017, the introduction of the Transformer architecture revolutionized natural language processing by replacing sequential recurrent neural networks (RNNs) and long short-term memory (LSTM) units with a fully parallelizable self-attention mechanism capable of handling entire token sequences simultaneously. This breakthrough mitigated vanishing gradient issues, facilitated effective modelling of long-range dependencies, and capitalized on the parallelism inherent in modern GPUs. Subsequent research revealed that scaling parameters, datasets, and computational resources directly cause the emergence of new abilities that are absent in smaller models. This scaling acts as a catalyst, triggering previously unseen capabilities and qualitative shifts in performance that smaller-scale models simply cannot achieve. This progression is exemplified by the GPT series: GPT-1 (117 million parameters) established the foundational viability of generative pre-training for natural language understanding. GPT-2 (1.5 billion parameters) demonstrated unprecedented zero-shot task transfer capabilities, generating coherent text across diverse domains without task-specific fine-tuning. GPT-3 (175 billion parameters) unveiled emergent few-shot and zero-shot reasoning abilities, proving that architectural scale alone could catalyse sophisticated cognitive capabilities. GPT-3.5 ( \approx 175 billion parameters) introduced conversational refinements and formed the foundation for ChatGPT’s November 2022 launch. GPT-4 ( \approx 1.8 trillion parameters) achieved multimodal understanding with text and image processing, released in March 2023. GPT-5 ( \approx 635 billion parameters for high reasoning) achieved complex tasks, deep analysis and multi-step reasoning, released in August 2025. Figure 3. Journey of GPT models 1.2. The Inference Bottleneck: Where Scale Meets Reality The hype around AI has focused on training, but the true technical challenge lies in inference, which is the point where AI systems operate in the real world. As models expand to hundreds of billions or even trillions of parameters, inference, the generation of outputs from trained models has emerged as the critical computational challenge. Real-world applications, including conversational agents, code generation tools, and content creation systems, necessitate high throughput (e.g., millions of tokens per second) and low latency (e.g., sub-second time-to-first-token and inter-token delays). Traditional monolithic inference pipelines, where tokenization, embedding, attention computations, feed-forward networks, and output processing occur within a single integrated runtime on uniform GPUs, are plagued by several issues: Figure 4. Where Scale Meets Reality • Resource Contention : The compute-bound prefill phase and memory-bound decode phase compete for the same GPU cores and memory bandwidth, causing significant underutilization with GPU utilization rates as low as 0.2\% in some cases. • Rigid Scaling : Addressing bottlenecks in one phase (e.g., attention) demands replication of the entire pipeline, leading to overprovisioning and inefficient resource allocation. • Fault Fragility : Failures in any component disrupt the whole inference process, creating vulnerable single points of failure that compromise system reliability. • Deployment Inflexibility : Optimizations in kernels or batching strategies require redeploying the complete system, impeding rapid iteration and development cycles. • Heterogeneous Workload Inefficiency : Uniform hardware struggles to optimally support both compute-intensive and memory-intensive tasks, with DRAM bandwidth saturation identified as the primary bottleneck in large-batch inference. These constraints highlight that conventional inference approaches cannot meet the demands of modern large language models, driving innovation toward specialized hardware architectures, modular pipeline designs, and co-optimized software-hardware solutions to overcome the inference bottleneck. 1.3. Microservices and Heterogeneous GPU Clusters: Where Modularity Meets Efficiency Figure 5. Where Modularity Meets Efficiency To overcome the limitations of monolithic inference systems, modern AI inference architectures have adopted microservices principles. This involves breaking down the inference pipeline into discrete, modular services such as Tokenization, Embedding, Prefill, Decode, and Post-Processing each independently scalable, monitored, and deployable. Orchestration platforms like Kubernetes enable seamless service discovery, load balancing, and rolling updates, providing resilience and allowing zero-downtime deployments. A cornerstone of this microservices paradigm is the strategic deployment of heterogeneous GPU clusters tailored to workload characteristics. Prefill services, which perform extensive matrix-matrix multiplications during initial context processing, are optimally hosted on compute-optimized GPUs such as NVIDIA A100 or H100, equipped with thousands of CUDA cores, specialized Tensor Cores, and multi-TB/s high-bandwidth memory (HBM) for maximum computational throughput. Decode services, focused on memory-bound sequential token generation through key-value (KV) cache operations—benefit significantly from memory-optimized GPUs