FailLite : Failure-Resilient Model Serving for Resource-Constrained Edge Environments
Li Wu, Walid A. Hanafy, Tarek Abdelzaher, David Irwin, Jesse Milzman, Prashant Shenoy · 2025
Read FailLite : Failure-Resilient Model Serving for Resource-Constrained Edge Environments by Li Wu, Walid A. Hanafy, Tarek Abdelzaher, David Irwin, Jesse Milzman, Prashant Shenoy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15856.txt.
Introduction
Li Wu and co-authors wrote a 2025 paper on FailLite, a system that keeps AI services running at the edge when a device fails. Cloud style backup, full copies of each model, is too heavy for edge boxes. Their system backs each model with a smaller variant instead.
Hot backups stand ready for vital apps, while cold backups wake on demand for the rest. A staged failover brings apps back in a smart order. On a test bed with twenty-seven models, every failed app recovered in well under a second on average. Accuracy dipped by less than one percent. The design suits clinics, factories, and other sites far from data centres.
Catalog record
| CREATOR | Li Wu, Walid A. Hanafy, Tarek Abdelzaher, David Irwin, Jesse Milzman, Prashant Shenoy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15856
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FailLite : Failure-Resilient Model Serving for Resource-Constrained Edge Environments Li Wu, Walid A. Hanafy, Tarek Abdelzaher, David Irwin, Jesse Milzman, Prashant Shenoy Abstract . Model serving systems have become popular for deploying deep learning models for various latency-sensitive inference tasks. While traditional replication-based methods have been used for failure-resilient model serving in the cloud, such methods are often infeasible in edge environments due to significant resource constraints that preclude full replication. To address this problem, this paper presents FailLite, a failure-resilient model serving system that employs (i) a heterogeneous replication where failover models are smaller variants of the original model, (ii) an intelligent approach that uses warm replicas to ensure quick failover for critical applications while using cold replicas, and (iii) progressive failover to provide low mean time to recovery (MTTR) for the remaining applications. We implement a full prototype of our system and demonstrate its efficacy on an experimental edge testbed. Our results using 27 models show that FailLite can recover all failed applications with 175.5ms MTTR and only a 0.6% reduction in accuracy. 1. Introduction In recent years, deep learning-based systems have revolutionized many domains, such as autonomous driving, augmented reality, personal assistants, and Internet of Things analytics ( Li et al. 2020 ; Satyanarayanan et al. 2021 ; Li et al. 2024 ) . The broad adoption of deep learning models is attributed to the availability of accelerators, such as Nvidia GPUs and Google TPUs. To train a Deep Neural Network (DNN) model, users often rely on cloud data centers (e.g., Amazon AWS and Google Cloud) to access their powerful DNN accelerators via simple application interfaces. For example, services like Amazon SageMaker allow users to upload their data and use a simple user interface to select the models to train ( Amazon Web Services 2025 ) . In contrast to DNN training, which is largely a batch workload, runtime use of trained models—often called model serving or model inference—tends to be latency-sensitive in nature. To meet the latency requirement of such applications, users often rely on edge data centers placed at the network edge ( Satyanarayanan et al. 2009 ; Satyanarayanan 2017 ) . The advances in edge accelerators (e.g., edge GPUs and TPUs) have made edge computing a popular choice for deploying latency-sensitive models serving applications ( Liang et al. 2023a ; Satyanarayanan et al. 2021 ) . With the advances in AI, researchers have addressed many challenges in building model serving systems. For instance, researchers have addressed the latency challenges by optimizing the network overheads ( Hanafy et al. 2023b ; Huang et al. 2020 ) , processing latency ( Crankshaw et al. 2017 ; Gujarati et al. 2020 ; Soifer et al. 2019 ; Zhang et al. 2019 ; Samplawski et al. 2020 ; Zhang et al. 2021 ; Liang et al. 2023a ) , cost ( Zhang et al. 2020b ; Ahmad et al. 2024a ; Liang et al. 2020 ; Fan and Ansari 2019 ) , energy efficiency ( Liang et al. 2023b ; Hanafy et al. 2021 ; Wan et al. 2020 ; Zadeh et al. 2020 ) , and addressing challenges of workload dynamics ( Ahmad et al. 2024b ; Ahmad et al. 2024a ; Zhang et al. 2020a ) . However, there is limited research on the failure resiliency of model serving systems, especially in resource-constrained environments, such as the edge. The traditional approach for ensuring fault tolerance of an application task is to replicate it on additional servers and to fail over to a backup server when the primary fails. This approach to crash fault tolerance is commonly used for critical cloud-based services. Cloud-based model serving, which is useful when tight latency requirements are not needed, has also employed replication to satisfy demand and latency constraints and to ensure failure resiliency. For instance, a recent effort ( Soifer et al. 2019 ) utilized backup requests to multiple replicas and cross-server cancellation to ensure that all requests are processed within their latency requirements. Which such replication may be feasible in cloud settings where resources are abundant and can be scaled on demand, edge environments are often highly resource constrained, which may preclude replication of deployed models for fault tolerance. Moreover, edge computing clusters are often deployed across multiple geographically distributed locations and may face many single points of failure issues, where the entire cluster becomes out of service or unreachable. Designing a failure-resilient model serving system at the edge involves several key challenges. First, such a system must address resource constraints where traditional replication and failover solutions may not be feasible. Second, the interactive nature of model serving requires low mean time to recovery (MTTR) in the event of a failure, such that users are minimally impacted. Third, the systems should be able to handle concurrent server failures or the failure of an edge location by restoring applications at other suitable edge sites. To address this problem, we present FailLite , a failure-resilient machine learning inference system for edge computing clusters. FailLite exploits several well-known accuracy-resource trade-offs for deep learning models, such as choosing the right-sized DNN model ( Hanafy et al. 2021 ; Ahmad et al. 2024b ; Zhang et al. 2020a ; Halpern et al. 2019 ; Ahmad et al. 2024a ; Li et al. 2023 ) and compression techniques, such as quantization and pruning ( Liu et al. 2020 ; Hashemi et al. 2017 ; Zadeh et al. 2020 ) , all of which ignificantly decrease resource requirements and latency with a minor reduction in accuracy. The core insight of FailLite is to provide fault tolerance using the concept of heterogeneous replication , where we use a smaller variant of the original model as a failover replica, ensuring high resiliency in such resource-constrained environments. Our second insight is that applications can vary in terms of their criticality and failure resilience needs, allowing FailLite to intelligently choose the level of failure resilience on a per-model basis. Specifically, FailLite can intelligently use proactive replication and warm replicas for the most important models, while using cold replicas with progressive failover for low downtimes for the remaining models. Such an approach can provide good accuracy upon failover despite using smaller models as replicas, while providing low mean time to recovery. In designing, implementing, and evaluating FailLite , our paper makes the following contributions: (1) We present the design of FailLite , a failure-resilient model serving system for resource-constrained environments. To our knowledge, FailLite is the first effort to tackle this issue. (2) We present a two-step failover approach that utilizes heterogeneous replication, considers applications’ criticality levels, and progressively loads cold backups to optimize their MTTR. (3) We implemented FailLite as a framework agnostic control plane with failure detection and failure-resilient policies. We integrated FailLite with the Nvidia Triton Inference Server, an open-source model serving framework to show the utility of our design. We also integrated our FailLite with a custom-built discrete event simulation platform for large-scale evaluations. (4) Our experimental evaluation of FailLite of a real test bed using 27 machine learning models shows that in resource constraint environments, FailLite can recover all failed applications under 175.5 ms MTTR only for 0.6% reduction in accuracy. In addition, our large-scale simulation