Characterizing Metastable Faults and Failures
Ali Farahbakhsh † , Qingjie Lu ∗ , Lorenzo Alvisi † , Andreas Haeberlen ∗ , Robbert Van Renesse † † Cornell University, ∗ University of Pennsylvania · 2026
Read Characterizing Metastable Faults and Failures by Ali Farahbakhsh † , Qingjie Lu ∗ , Lorenzo Alvisi † , Andreas Haeberlen ∗ , Robbert Van Renesse † † Cornell University, ∗ University of Pennsylvania (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00942.txt.
Introduction
Farahbakhsh and co-authors give a causal account of metastable failures in this 2026 systems paper. A metastable failure is a bad state a system sustains on its own. No attacker or harsh input is needed to keep it going. Past work tracked symptoms and cast such failures as self-sustaining overload. Causes stayed unknown, and not all cases look like overload.
The team traces these failures to metastable faults. A fault is a cycle of interactions among parts that destabilizes the whole. Each part alone tends to stabilize. Scheduling choices decide which tendency wins. When the destabilizing cycle gains the upper hand, the failure locks in. From this account they derive a method to predict such failures. They also sketch how to build systems that tolerate the faults. Three case studies test the method's reach. The paper reframes a feared outage class as a structure that design can hunt early.
Catalog record
| CREATOR | Ali Farahbakhsh † , Qingjie Lu ∗ , Lorenzo Alvisi † , Andreas Haeberlen ∗ , Robbert Van Renesse † † Cornell University, ∗ University of Pennsylvania |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00942
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Characterizing Metastable Faults and Failures Ali Farahbakhsh † , Qingjie Lu ∗ , Lorenzo Alvisi † , Andreas Haeberlen ∗ , Robbert Van Renesse † † Cornell University, ∗ University of Pennsylvania Abstract . Metastable failures are hard to detect, prevent, and mitigate. During a metastable failure, a system exhibits self-sustaining bad behavior even in the absence of adversarial conditions. Prior work focuses on symptoms and has portrayed metastable failures as instances of self-sustaining overload. This characterization leaves the underlying failure causes and dynamics unknown, and does not account for metastable failures that do not manifest as overload. We present the first causal characterization of metastable failures by identifying their origin in metastable faults , i.e. , structural destabilizing cycles of interaction among systems components that, in isolation, are stabilizing. Metastable failures arise when scheduling decisions let these destabilizing interactions gain the upper hand over the individual components’ stabilizing tendencies. We then derive a methodology to predict metastable failures, and to build metastable-fault-tolerant (MFT) systems. We apply our methodology to three case studies, showcasing the generality of our results. 1. Introduction Metastable failures ( 30 ; 40 ; 19 ; 6 ; 18 ; 21 ; 45 ; 44 ; 46 ; 27 ) have drawn significant attention in the systems community as a particularly challenging failure mode—one in which a system exhibits self-sustaining bad behavior after a finite shock ( 5 ; 26 ; 28 ; 16 ; 23 ; 3 ; 7 ) . For instance, a temporary surge in client load might lead to persistently low goodput even after the load goes back to normal. The system does not control the shock; however, the ensuing bad behavior is due solely to how the system works. Such failures are costly ( 26 ) , hard to detect or predict ( 6 ) , and difficult to mitigate ( 21 ; 45 ; 44 ; 46 ) , making it critical to understand their underlying causes. The prevailing characterization of metastable failures is mostly phenomenological : it starts from symptoms—high latency and low goodput under normal conditions—and equates metastability with self-sustaining overload ( 5 ; 26 ) . It then identifies the culprit as either an internal load amplification or capacity degradation ( 26 ) . By focusing on symptoms, this approach fails to fully explain the underlying causes that give rise to metastable failures, leaving us vulnerable in two ways: ( i ) when these failures occur, we lack targeted remedies—reports repeatedly show that only drastic measures such as restarting the system or disabling communication can halt a live failure ( 21 ; 45 ; 44 ; 46 ; 27 ) ; and ( ii ) we risk overlooking metastable failures that do not manifest with the same symptoms. To move beyond these limitations, we return to a fundamental principle of fault tolerance: every failure stems from a fault . We present the first analytical characterization of metastable failures that reveals the structural faults that cause them—henceforth called metastable faults. Metastable faults are sins of composition : they capture circular destabilizing interactions among individually stabilizing components. Each component responds to a shock by trying to stabilize locally, but in doing so destabilizes others, coupling them in perpetual destabilization and producing self-sustaining bad behavior. This characterization also explains the usual symptoms of high latency and low goodput: resources that in the stable state would be spent in useful work are instead continuously wasted in futile attempts to stabilize. Building on our experience studying various metastable failures, we outline a methodology for predicting such failures and designing metastable-fault-tolerant (MFT) systems. Our methodology revolves around a proof outline for showing that a system is MFT, i.e. , that the system avoids metastable failures despite harboring a metastable fault. We also introduce a domain-specific language (DSL), Nyx, to ( i ) reproduce said failures in vitro and ( ii ) assist designers in applying the methodology. Nyx comes with a toolkit including an interpreter and a tool to visualize metastable failures. Our methodology relies on a key insight: metastable failures emerge when poor scheduling favors the fault over stabilizing interactions. In the absence of stabilizing interactions, perpetual destabilization can be trivially inevitable. Metastable failures occur instead when the components can help each other stabilize, but any stabilization is interrupted by some destabilizing event. The reason is a fatal coupling between destabilization and poor scheduling: the scheduler favors the fault’s destabilizing tendencies, pushing the system away from stability, creating in turn more destabilizing interactions, prompting the system to make more bad scheduling decisions— ad infinitum . Based on this insight, our methodology augments compositions of stabilizing components with scheduling decisions that make them MFT. These decisions must: ( i ) schedule destabilizing interactions tentatively, and ( ii ) defer destabilizing interactions until stabilizing ones have achieved global stability. The latter ensures eventual global stability; the former ensures that, once achieved, stability is never lost. We prove metastable fault tolerance with respect to specific adversaries. Faults are notoriously difficult to locate ( 6 ; 30 ) , and often require post-mortem analysis. However, these faults are triggered by an adversary. Therefore, proving stabilization despite an adversary imposing shocks on the system establishes tolerance against all faults that the adversary can trigger—a single proof covers a wide range of faults. We envision research on metastable faults and failures to resemble security research: new adversaries will expose new vulnerabilities. We have applied our methodology to three case studies: ( i ) the retry storm ( 21 ; 45 ; 44 ; 46 ; 41 ) , which serves as the running example for our characterization, ( ii ) a novel case study from a major commercial gaming platform with hundreds of millions of users, where metastability manifests as oscillation, and ( iii ) the look-aside cache incident ( 5 ; 26 ) , used to demonstrate the generality of our approach. For the second case study, we use an internal post-mortem of an incident involving a cluster manager in one of the platform’s data centers. We design a new cluster manager and prove it MFT. The fault in this incident is subtle and hard to locate without post-mortem insight, yet we show that fault tolerance is achievable without prior knowledge of the fault. The third case study instead illustrates that sometimes one can simply remove the fault, thereby eliminating also the failure. Our methodology is a modest first step towards developing and deploying MFT software in production. Metastable faults span the entire stack ( 6 ; 30 ) , and the vast scale of production systems makes locating them even more challenging. As a result, while the community has developed robust toolboxes to prevent, detect, and mitigate other modes of failure, e.g. , deadlocks ( 10 ; 24 ; 48 ) , we lack an equivalent toolbox for metastable failures. Our characterization lays the foundation for such a pursuit by identifying the underlying principles, and our methodology demonstrates the practical utility of this characterization. In short, we make the following contributions: • We introduce metastable faults, and show how together with poor scheduling they lead to metastable failures; • we present a methodology for predicting metastable