C Snake : Detecting Self-Sustaining Cascading Failure via Causal Stitching of Fault Propagations Conference: European Conference on Computer Systems; April 27–30, 2026; Edinburgh, Scotland, UK European Conference on Computer Systems (EUROSYS ’26), April 27–30, 2026, Edinburgh, Scotland, UK DOI: 10.1

Shangshu Qian, Lin Tan, Yongle Zhang · 2025

Read C Snake : Detecting Self-Sustaining Cascading Failure via Causal Stitching of Fault Propagations Conference: European Conference on Computer Systems; April 27–30, 2026; Edinburgh, Scotland, UK European Conference on Computer Systems (EUROSYS ’26), April 27–30, 2026, Edinburgh, Scotland, UK DOI: 10.1 by Shangshu Qian, Lin Tan, Yongle Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-26529.txt.

Introduction

Shangshu Qian, Lin Tan, and Yongle Zhang present CSnake, a testing tool that exposes self-sustaining failure chains in distributed systems. Their paper was published in 2025. In such a chain, one fault triggers the next, and the outage keeps itself going across many machines.

Normal fault testing misses these chains, because each step needs its own specific trigger. CSnake links single fault tests together by cause, comparing each test run with a clean run to see which new faults a fault sets off. A budget plan tests the most varied faults first, and a quick check avoids joining steps whose conditions clash. The tool found fifteen such bugs in five systems, and several were confirmed or fixed.

Catalog record

CREATORShangshu Qian, Lin Tan, Yongle Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE139 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-26529

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

C Snake : Detecting Self-Sustaining Cascading Failure via Causal Stitching of Fault Propagations Conference: European Conference on Computer Systems; April 27–30, 2026; Edinburgh, Scotland, UK European Conference on Computer Systems (EUROSYS ’26), April 27–30, 2026, Edinburgh, Scotland, UK DOI: 10.1145/3767295.3769373 ISBN: 979-8-4007-2212-7/26/04 CCS: Computer systems organization Cloud computing CCS: Software and its engineering Software testing and debugging Shangshu Qian, Lin Tan, Yongle Zhang Abstract . Recent studies have revealed that self-sustaining cascading failures in distributed systems frequently lead to widespread outages, which are challenging to contain and recover from. Existing failure detection techniques struggle to expose such failures prior to deployment, as they typically require a complex combination of specific conditions to be triggered. This challenge stems from the inherent nature of cascading failures, as they typically involve a sequence of fault propagations, each activated by distinct conditions. This paper presents C Snake , a fault injection framework to expose self-sustaining cascading failures in distributed systems. C Snake uses the novel idea of causal stitching , which causally links multiple single-fault injections in different test workloads to simulate complex fault propagation chains. To identify propagation chains between faults, C Snake designs a counterfactual causality analysis of fault propagations – fault causality analysis (FCA): FCA compares the execution trace of a fault injection run with its corresponding profile run (i.e., running the same test without the injection) and identifies any additional faults triggered, which are considered to have a causal relationship with the injected fault. To address the large search space of fault and workload combinations, C Snake employs a three-phase allocation (3PA) protocol of test budget that prioritizes faults with unique and diverse causal consequences, thereby increasing the likelihood of uncovering conditional fault propagations. Furthermore, to avoid incorrectly connecting fault propagations from workloads with incompatible conditions, C Snake performs a local compatibility check that approximately checks the compatibility of the path constraints associated with connected fault propagations with low overhead. C Snake has detected 15 bugs that resulted in self-sustaining cascading failures in five widely deployed distributed systems, five of which have been confirmed with two fixed. † † cc-license: by 1. Introduction Distributed systems are designed to tolerate component failures ( Chandra and Toueg 1996 ) . However, recent research has revealed self-sustaining cascading failures ( Li et al. 2018 ; Huang et al. 2022 ; Qian et al. 2023 ; Guo et al. 2013 ) , wherein a fault propagates across components and replicates itself through a self-reinforcing loop ( Huang et al. 2022 ; Qian et al. 2023 ) . These failures undermine the intended fault tolerance, resulting in widespread outages with severe consequences. For example, a self-sustaining cascading failure happened in Amazon AWS ( AWS [n. d.] ) on July 30th, 2024, significantly disrupting core AWS services, including AWS Lambda, EC2, and S3. The incident also brought down many dependent services, such as Whole Foods Supermarket, Amazon Alexa, and Goodreads ( Harris 2024 ) . Figure 1 . A real-world self-sustaining cascading failure from Amazon AWS. The Cell (Cluster) Manager manages a cluster of hosts (servers), each storing a set of data shards. Such failures often require intricate combinations of conditions to manifest ( Qian et al. 2023 ) , as they typically involve a sequence of fault propagations, each activated by distinct conditions. For instance, as shown in Figure 1 , the Amazon AWS incident ( AWS [n. d.] ) occurred in a cluster of servers (hosts) managed by a cluster (cell) manager, with each server hosting a set of data shards. During a routine system upgrade, the cell manager ( 1 ) restarted some hosts (e.g., Host 1), and ( 2 ) redistributed their shards to other hosts (e.g., Host 2). Due to a latent bug in the load balancer, the cell manager mistakenly redistributed all low-throughput shards to a small number of hosts (Host 2). Consequently, these hosts started ( 3 ) sending abnormally large status reports that include metadata for all hosted shards. The increased size of these reports caused delays in both transmission and processing ( 4 ), causing the management system to misclassify these servers as unhealthy. The management system consequently removed these servers from the cluster, and ( 5 ) redistributed their shards to other servers, which causes the receiving servers (e.g., Host 3) to send large-sized reports ( 6 ) and subsequently be removed from the cluster ( 7 ). This failure involves two fault propagations, each requiring distinct conditions. First, for a server removal to propagate and trigger incorrect redistribution ( 1 \rightarrow 2 and 4 \rightarrow 5 ), the redistributed shards must have low throughput. Second, for incorrect redistribution to induce report delay and subsequent server removal ( 2 \rightarrow 4 and 5 \rightarrow 7 ), the report size (or, the number of affected shards) must be sufficiently large. Existing fault injection techniques ( Wu et al. 2024 ; Li et al. 2018 ; Gao et al. 2023 ; Joshi et al. 2011 ; Natella et al. 2016 ; Marinescu and Candea 2009 ; Alquraan et al. 2018 ; Alvaro et al. 2015 ; Chen et al. 2020 ; Gunawi et al. 2011 ; Ju et al. 2013 ; Lu et al. 2019 ; Mohan et al. 2018 ; Li et al. 2019 ; Sun et al. 2022 ; Meiklejohn et al. 2021 ; Zhang et al. 2021a ; Chen et al. 2023 ; jep nda ) fall short of exposing such self-sustaining cascading failures before deployment, because they only inject faults into a limited set of manually selected (e.g., stress tests) or synthetically crafted workloads (e.g., benchmark workloads), which often lack the required combination of conditions . Conversely, if a workload satisfying both of the previously described conditions had been exercised, a traditional fault commonly used in existing fault injection frameworks – a server crash – would have exposed the AWS self-sustaining cascading failure before deployment. Missing triggering conditions is a known challenge to effective fault injection testing ( Li et al. 2021 ; Chen et al. 2023 ) . Without prior knowledge of the targeted bug, it is exceptionally hard for the developers to manually create test workloads or rules for workload generators ( Kiskis and Shin 1996 ; Han et al. 1995 ; Marinescu and Candea 2009 ; Joshi et al. 2011 ) that meet all necessary conditions. To tackle this challenge, we propose Causal Stitching to simulate complex fault propagation under complex conditions by causally linking multiple single-fault injections in different workloads . Each injection uncovers one step in the fault propagation chain. Our insight is that triggering one step in the chain requires less stringent conditions than triggering the whole chain, increasing the likelihood of achieving this through fault injection into existing workloads, such as the integration tests shipped with the system. To illustrate, consider the two faults involved in the Amazon AWS self-sustaining cascading failure: node loss ( f_{1} ) and imbalanced shard redistribution ( f_{2} ). Their causal relationships can be identified through separate fault injection runs. Specifically, in a test case t_{1} with low-throughput shards (condition c_{1} ), injecting f_{1} results in the triggering of an additional fault f_{2} , establishing the causal relationship f_{1}\rightarrow f_{2} . Conversely,

Read the complete text