Towards Resiliency in Large Language Model Serving with KevlarFlow
Shangshu Qian, Kipling Liu, P. C. Sruthi, Lin Tan, Yongle Zhang · 2026
Read Towards Resiliency in Large Language Model Serving with KevlarFlow by Shangshu Qian, Kipling Liu, P. C. Sruthi, Lin Tan, Yongle Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22438.txt.
Introduction
Shangshu Qian and four co-authors make model serving bounce back fast when hardware fails. Their 2026 preprint, KevlarFlow, notes that big clusters fail often, and one fault can down a whole service. Old recovery can take ten minutes, as gear restarts and huge weights reload.
KevlarFlow aims to cut that dead time to a small part of the old span. Fast recovery keeps chat and search tools live for users. It also saves the vast cost of idle chips. The work is a shield for the serving layer. It treats faults as routine, not rare.
Catalog record
| CREATOR | Shangshu Qian, Kipling Liu, P. C. Sruthi, Lin Tan, Yongle Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22438
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Resiliency in Large Language Model Serving with KevlarFlow Shangshu Qian, Kipling Liu, P. C. Sruthi, Lin Tan, Yongle Zhang Abstract Large Language Model (LLM) serving systems remain fundamentally fragile, where frequent hardware faults in hyperscale clusters trigger disproportionate service outages in the software stack. Current recovery mechanisms are prohibitively slow, often requiring up to 10 minutes to reinitialize resources and reload massive model weights. We introduce KevlarFlow , a fault tolerant serving architecture designed to bridge the gap between hardware unreliability and service availability. KevlarFlow leverages 1) decoupled model parallelism initialization, 2) dynamic traffic rerouting, and 3) background KV cache replication to maintain high throughput during partial failures. Our evaluation demonstrates that KevlarFlow reduces mean-time-to-recovery (MTTR) by 20x and, under failure conditions, improves average latency by 3.1x, 99th percentile (p99) latency by 2.8x, average time-to-first-token (TTFT) by 378.9x, and p99 TTFT by 574.6x with negligible runtime overhead in comparison to state-of-the-art LLM serving systems. 1 Introduction LLMs have become an essential component of modern digital infrastructure, demonstrating exceptional capabilities across a wide range of tasks. However, despite their ubiquity, the infrastructure supporting LLM serving remains fundamentally fragile ( Chu et al. 2025 ) . Operational data reveals that outages are a common occurrence even for industry leaders, with significant service interruptions reported for platforms such as OpenAI ( Lu 2024 ) and Anthropic’s Claude ( Bai 2025 ) . As these models are increasingly integrated into critical applications, the reliability of the underlying serving systems has emerged as a paramount concern. Figure 1 : Rolling average and p99 TTFT comparison between a typical LLM serving framework (baseline) and KevlarFlow under 2 requests per second (RPS). The red line indicates the time when one node fails. Y-axis is in log scale. The root cause of this reliability crisis is threefold. First, hardware failures are common in production‑level GPU clusters ( Cui et al. 2025 ; Grattafiori et al. 2024 ) and frequently lead to program crashes ( Tiwari et al. 2015 ) . Second, the software stack lacks robust fault tolerance mechanisms, further exacerbating the hardware instability. Modern LLM serving frameworks (e.g., vLLM ( Kwon et al. 2023 ) and TensorRT-LLM ( Nvidia ) ) rely on collective communication libraries such as Message Passing Interface (MPI) ( MPIForum 1994 ) and NVIDIA Collective Communication Library (NCCL) to implement complex model parallelism strategies ( Shoeybi et al. 2019 ) (e.g., tensor and pipeline parallelism). Such communication libraries often optimize for throughput instead of fault tolerance ( Hu et al. 2025 ; Weingram et al. 2023 ) and create complex inter-dependencies among participating nodes since the initialization of the serving program. As a result, the failure of a single node will render all other model parallelism participants, while still alive, unserviceable for user requests. Figure 1 (blue lines) shows the rolling average and p99 TTFT of a typical LLM serving cluster under node failures. When one node fails, incoming requests pile up the queue quickly due to reduced serving capacity, increasing the TTFT and affecting user experience. Third, slow recovery of LLM serving systems compounds these software and hardware issues by prolonging service degradation after a failure. Restoring a failed instance typically involves 1) a complete re-initialization of serving node (e.g., one instance on Amazon EC2), and 2) the reloading of massive model weights from remote storage. This process can take as long as 10 minutes ( Jaiswal et al. 2025b ) , and the LLM service (partially) backed by the failed node will remain unresponsive during the entire recovery time. Furthermore, widely used redundancy techniques from traditional distributed systems, such as data replication, have not been seamlessly adapted for the memory-intensive and state-heavy nature of LLM inference ( Strati et al. 2024 ) . While replication ensures high availability in databases, applying it to LLM serving remaining a challenging problem due to the unique constraints of GPU memory and the transient nature of the inference state. To address these challenges, we introduce KevlarFlow , a fault tolerant serving architecture designed to bridge the gap between hardware unreliability and service availability. To solve the hardware instability, KevlarFlow draws insight from the inherent redundancy found within the load balancer groups of serving systems ( Jain et al. 2025 ) . By treating the cluster not as a collection of rigid, independent instances but as a flexible pool of resources, KevlarFlow enables high-throughput serving that degrades gracefully rather than failing catastrophically. KevlarFlow achieves resiliency in LLM serving through three novel design contributions: 1. Dynamic Traffic Rerouting and Partial Availability. KevlarFlow introduces a traffic management strategy that handles partial model parallelism failures – scenarios where only a subset of nodes in a model parallel group fails. Instead of marking the entire instance as offline, KevlarFlow dynamically reroutes user traffic around the failed node to other healthy instances in the load balancing group. Crucially, it keeps the remaining healthy parts of the damaged model in service, allowing failed nodes to be replaced in the background. This approach significantly improves system throughput under failure conditions by preventing the waste of functional GPU resources blocked by dependencies. 2. Decoupled Model Parallelism Initialization. Unlike other frameworks, KevlarFlow decouples the initialization of model parallelism communicator from model weight loading, instead of performing both tasks at the start of the service. This architectural separation allows for dynamic reconfigurations of the parallelism schemes at runtime. If a node fails, the system can re-establish communication among the remaining healthy nodes without incurring the heavy overhead of a full restart or weight reload. 3. Background KV Cache Replication. To mitigate the latency penalty of request restarting, KevlarFlow implements a low-overhead background replication mechanism for the KV cache. By replicating the intermediate inference state to other nodes in the load balancing group during runtime, KevlarFlow ensures that if a failure occurs, partially served requests can be continued near-instantly on a live node. This replication converts what would be a hard failure requiring a restart into a seamless migration, preserving the user’s session context. Our evaluation demonstrates that KevlarFlow effectively masks hardware failures with minimal performance penalties: 1. Under partial failure scenarios, KevlarFlow improves average latency by 3.1x, p99 latency by 2.8x, average TTFT by 378.9x, and p99 TTFT by 574.6x compared to failure behaviors in state-of-the-art LLM serving frameworks. 2. KevlarFlow shortens the mean-time-to-recovery (MTTR) of node failures from 10 minutes ( Jaiswal et al. 2025b ) to 30 seconds, a 20x improvement. 3. The background KV cache replication incurs negligible overhead during normal operations, highlighting KevlarFlow ’s potential as an “always-on” fault tolerance solution for the next-generation AI infrastructure. KevlarFlow is the first LLM serving framework that can tolerate runtime node failures, allowing non-interruptive request handling during failure recovery. 2 Related Work In this section, we introduce existing fault tolerance mechanisms in both LLM training and serving applications, and show that KevlarFlow is the only approach that can tolerate node failures in serving applications. Fault-Tolerant LLM Training. Communication and computation patterns are fundamentally different in LLM training and serving applications. Fault tolerance training techniques ( Gandhi et al. 2024 ; Duan et al. 2024 ) do not directly apply to serving for two reasons: 1) serving maintains transient states (i.e., KV cache) for each