Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI
Saicharan Kolluru · 2025
Read Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI by Saicharan Kolluru (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17593.txt.
Introduction
Saicharan Kolluru benchmarks the two leading open-source systems for serving large language models, vLLM and HuggingFace's Text Generation Inference, head to head. Production deployments must balance throughput, latency, and memory use, and the right choice depends on the workload. Kolluru measures both systems across those dimensions, including scalability, using LLaMA-2 models from seven to seventy billion parameters.
Under high-concurrency workloads, vLLM delivered up to twenty-four times the throughput of its rival, thanks to its PagedAttention memory mechanism. The rival system, however, showed lower tail latencies for interactive, single-user scenarios, where responsiveness matters more than volume. Kolluru closes with useful selection guidance: high-throughput batch processing points one way, latency-sensitive interactive applications with moderate concurrency the other. The study gives deployment teams measured profiles instead of folklore.
Catalog record
| CREATOR | Saicharan Kolluru |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-17593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Comparative Analysis of Large Language Model Inference Serving Systems: A Performance Study of vLLM and HuggingFace TGI Saicharan Kolluru Abstract The deployment of Large Language Models (LLMs) in production environments requires efficient inference serving systems that balance throughput, latency, and resource utilization. This paper presents a comprehensive empirical evaluation of two prominent open-source LLM serving frameworks: vLLM and HuggingFace Text Generation Inference (TGI). We benchmark these systems across multiple dimensions including throughput performance, end-to-end latency, GPU memory utilization, and scalability characteristics using LLaMA-2 models ranging from 7B to 70B parameters. Our experiments reveal that vLLM achieves up to 24x higher throughput than TGI under high-concurrency workloads through its novel PagedAttention mechanism, while TGI demonstrates lower tail latencies for interactive single-user scenarios. We provide detailed performance profiles for different deployment scenarios and offer practical recommendations for system selection based on workload characteristics. Our findings indicate that the choice between these frameworks should be guided by specific use-case requirements: vLLM excels in high-throughput batch processing scenarios, while TGI is better suited for latency-sensitive interactive applications with moderate concurrency. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, from conversational AI to code generation and content creation [ 1 , 2 , 3 ] . However, the deployment of these models in production environments presents significant engineering challenges. The computational demands of autoregressive text generation, combined with the massive parameter counts of modern LLMs, necessitate specialized serving infrastructure that can efficiently manage GPU resources while meeting application-specific performance requirements. The serving infrastructure for LLMs must address several competing objectives: maximizing throughput to serve many concurrent users, minimizing latency for responsive user experiences, and efficiently utilizing expensive GPU resources. Different applications prioritize these objectives differently—a chatbot requires low latency for individual requests, while a batch document processing system prioritizes throughput. This variation in requirements has led to the development of specialized serving frameworks, each making different design trade-offs. Among the available open-source solutions, vLLM [ 4 ] and HuggingFace Text Generation Inference (TGI) [ 5 ] have emerged as leading frameworks, widely adopted in both research and production settings. vLLM introduces PagedAttention, a novel attention mechanism inspired by virtual memory paging in operating systems, which dramatically reduces memory fragmentation and enables higher batch sizes. TGI, developed by HuggingFace, focuses on production-grade deployment features including distributed inference, quantization support, and extensive model compatibility. Despite their popularity, there exists limited systematic comparison of these frameworks across realistic deployment scenarios. Existing benchmarks often focus on narrow aspects of performance or use synthetic workloads that may not reflect production traffic patterns. Furthermore, the performance characteristics can vary significantly based on model size, hardware configuration, and request patterns, making it challenging for practitioners to make informed deployment decisions. 1.1 Contributions This paper makes the following contributions: 1. We present a comprehensive empirical evaluation of vLLM and TGI across multiple LLaMA-2 model sizes (7B, 13B, and 70B parameters) on modern GPU hardware. 2. We analyze performance across diverse workload patterns including varying concurrency levels, prompt lengths, and generation lengths, reflecting realistic deployment scenarios. 3. We provide detailed measurements of throughput, latency distributions, GPU memory utilization, and scheduling efficiency under different configurations. 4. We identify key architectural differences that explain observed performance characteristics and discuss their implications for system design. 5. We offer practical recommendations for framework selection based on application requirements and workload characteristics. The remainder of this paper is organized as follows: Section 2 reviews related work on LLM serving systems. Section 3 describes our experimental methodology. Section 4 presents detailed performance results across multiple dimensions. Section 5 analyzes the architectural factors underlying observed differences. Section 6 discusses practical implications, and Section 7 concludes. 2 Background and Related Work 2.1 LLM Inference Challenges Serving LLMs for text generation differs fundamentally from traditional model inference due to the autoregressive nature of generation. Each token is generated sequentially, with the model conditioning on all previously generated tokens. This creates several challenges: Memory Management: The key-value (KV) cache, which stores attention states for previously generated tokens, grows linearly with sequence length and can consume significant GPU memory. For a LLaMA-70B model generating 2048 tokens, the KV cache alone requires approximately 140GB of memory per request. Batching Complexity: Unlike computer vision models where batching is straightforward, LLM requests complete at different times due to varying generation lengths. Naive batching approaches waste computation on padding or limit batch sizes significantly. Memory Fragmentation: Traditional serving systems pre-allocate contiguous memory for the maximum possible sequence length, leading to significant fragmentation and underutilization when actual sequences are shorter. Scheduling: Balancing throughput and latency requires sophisticated scheduling that can dynamically batch requests while maintaining fairness and meeting service-level objectives. 2.2 LLM Serving Systems Several frameworks have been developed to address these challenges: vLLM introduces PagedAttention, which manages the KV cache using non-contiguous memory blocks similar to virtual memory systems. This approach eliminates memory fragmentation and enables near-optimal memory utilization. vLLM also implements continuous batching, which dynamically adds new requests to the batch as ongoing generations complete. HuggingFace TGI provides a production-ready serving solution with features including model parallelism, quantization, speculative decoding, and extensive model support. TGI emphasizes ease of deployment and integration with the HuggingFace ecosystem. TensorRT-LLM (NVIDIA) offers highly optimized inference through kernel fusion, INT8/FP8 quantization, and multi-GPU parallelism, but requires model-specific optimization and compilation. DeepSpeed-FastGen focuses on dynamic splitting techniques to improve throughput for variable-length generation workloads. Other systems include FasterTransformer, Triton Inference Server, and Ray Serve, each with different design philosophies and optimization strategies. 2.3 Benchmarking Studies While several benchmarking efforts exist, they often have limited scope. Prior work has examined individual aspects such as memory efficiency or single-model performance, but comprehensive comparisons across multiple dimensions remain scarce. Our work extends previous efforts by providing systematic evaluation across model sizes, workload patterns, and performance metrics relevant to production deployment decisions. 3 Methodology 3.1 Experimental Setup 3.1.1 Hardware Configuration All experiments were conducted on NVIDIA A100 GPUs (80GB) to ensure sufficient memory for large models. We used the following infrastructure: • GPU: 4x NVIDIA A100 80GB (SXM4) • CPU: AMD EPYC 7763 64-Core Processor • RAM: 512GB DDR4 • Storage: NVMe SSD for model loading • Network: 100Gbps interconnect 3.1.2 Software Versions • vLLM v0.6.1 • HuggingFace TGI v2.3.0 • CUDA 12.1 • PyTorch 2.1.0 • Model: LLaMA-2-7B, 13B, and 70B 3.2 Models and Configurations We evaluated three LLaMA-2 model variants: 1. LLaMA-2-7B: 7 billion parameters, single GPU deployment 2. LLaMA-2-13B: 13 billion parameters, single GPU deployment 3. LLaMA-2-70B: 70 billion parameters, tensor parallel across 4 GPUs For the 70B model, we used tensor parallelism to distribute the model across multiple GPUs. Both frameworks were configured with FP16 precision to balance performance and quality. 3.3 Workload Characteristics We designed benchmark workloads to reflect realistic deployment scenarios: 3.3.1 Request Patterns • Low Concurrency (1-10 users): Simulates interactive chatbot scenarios • Medium Concurrency (10-50 users): Typical production load • High Concurrency (50-200 users): Stress test for peak demand 3.3.2 Input/Output Characteristics • Prompt lengths: Sampled from Poisson distribution (mean=512 tokens) • Generation lengths: Sampled from Poisson distribution (mean=256 tokens) • Temperature: 0.7 with top-p sampling (p=0.9) 3.3.3 Dataset We used prompts from the ShareGPT dataset, which contains real user-AI conversations, ensuring