Explaining Failures of Cyber-Physical Systems with Actual Causality
Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler, Moshe Y. Vardi · 2026
Read Explaining Failures of Cyber-Physical Systems with Actual Causality by Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler, Moshe Y. Vardi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24546.txt.
Introduction
Elimelech and colleagues bring actual causality to the practical problem of explaining why an autonomous system failed. Self driving cars and other cyber physical systems, especially those with neural parts, cannot be fully verified before launch. Failures will happen, and someone must explain them. Actual causality, a formal framework for spotting which events genuinely caused an outcome, had previously served only simple systems, such as image classifiers.
Theory gaps blocking its use on cyber physical systems are identified and closed. Guidance for deriving correct explanations in that domain follows. Two new system agnostic algorithms result. One prioritises the best explanations. The other prioritises derivation efficiency, so users can choose their trade off. The approach is black box friendly, working from traces and specs rather than source code. Elimelech's 2026 team argues trustworthy autonomy needs post failure explanation as a first class output, not an afterthought.
Catalog record
| CREATOR | Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler, Moshe Y. Vardi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24546
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Explaining Failures of Cyber-Physical Systems with Actual Causality Khen Elimelech, Tom Yaacov, David A. Kelly, Hana Chockler, Moshe Y. Vardi Abstract Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, face increasingly complex demands, and yet are expected to act reliably. The black-box nature often characterizing such systems, especially those relying on neural components, makes it impossible to fully verify the system behavior prior to deployment. Unfortunately, unexpected failures—cases when the system does not comply with its specification—are inevitable and may have catastrophic implications. To improve trust in the system and facilitate future mitigation after a failure occurs, it is important to try to derive an explanation for the unexpected system behavior. This paper introduces the novel concept of leveraging the framework of actual causality for CPS failure explanation. Up until now, this framework was only used to derive explanations in the context of simple systems, such as image classifiers. This paper addresses the theoretical gaps and provides the guidance needed to allow for correct explanation derivation in the CPS domain. Beyond the theoretical contribution, the paper presents two novel, practical, system-agnostic explanation derivation algorithms, allowing to prioritize either explanation optimality or derivation efficiency. The approach is demonstrated and evaluated in the context of a neural-network-controlled autonomous car, designed to avoid collisions. I Introduction I-A Background: Explaining Failures Cyber-Physical Systems (CPSs) rely on algorithms and digital computation to control real-world machines, coordinating their components to demonstrate desired behaviors [ 20 , 21 ] . Modern CPSs, such as self-driving cars or assistive robots, are expected to achieve complex goals autonomously, i.e., without human involvement, and robustly, i.e., without failure, in a variety of complex, previously unseen environments. With the growing demands such systems have to face, their internals also tend to grow in complexity—with modern systems almost ubiquitously leveraging “black-box” Deep Neural-Network (NN)-based components, such as controllers and perceptions modules, to meet these demands. The complexity and opaqueness of these systems, together with intractable environment variability, often make it that systems can only be tuned empirically, making it impossible to fully guarantee their behavior ahead of deployment. Unfortunately, failure cases, i.e., when the system does not comply with its given specification, are inevitable, and unexpected failures can lead to catastrophic implications. Therefore, when a system is deployed and a failure is encountered, it is important to derive an explanation for the unexpected system behavior, to help us to avoid, anticipate, or better mitigate similar failures in the future. This process may also take place proactively , prior to the system deployment, as part of its formal verification process; in that case, we may first smartly search for failure scenarios in simulation—a problem known as falsification [ 9 ] —and, if and when those are found, derive their underlying explanations. Explaining the reasons for failure allows us to better understand system vulnerabilities and provide (some) guarantees for its behavior, improve trust in the system, and provide guidelines for system improvements. The importance of finding explanations is even more significant when the system components are well-tuned (in the NN case, -trained), making failures rarer and less expected. Understanding the reasons for events and deriving explanations can be done through causal analysis [ 14 ] . We should note that the prevalent framework for causality nowadays, which the reader is more likely to be familiar with, is that of general or type causality [ 19 ] . This framework is concerned with deriving forward-looking, often probabilistic, causal models, based on statistical analysis of prior experiences, for the purpose of prediction. This framework is not suitable for our objective of explaining specific (and, possibly, rare) failures. In contrast, the study of actual causality [ 15 ] is backward-looking, and can help us build causal models and derive causal explanations underlying specific events in the past—in alignment with our interests. The framework is of aid when there are numerous potential causes for an event, whose involvement in the outcome needs to be established—as is often the case in CPS, where multiple environmental factors may affect the system’s decisions. The high-level goal of this work is to deploy the framework of actual causality for the purpose of deriving causal explanations of failure events of (black-box) autonomous CPSs . I-B Contribution This paper’s contribution is first and foremost conceptual: we introduce the idea of explaining CPS failures through the lens of actual causality . To support this, the main objective and contribution of this paper is applying the framework of actual causality to construct explanations of CPS behaviors and failures, and extending it as necessary . The framework thus far has only been practically applied in the context of black-box “classifier” models (e.g., in [ 5 , 7 ] ), and not in temporally-extended models, as in the CPS case. We start by singling out the differences between the standard “classifier” case and the CPS and demonstrating why naive attempts to use the framework, without addressing these differences, may lead to the derivation of incorrect explanations. We then provide the correct guidelines for explanation derivation in the CPS case. As an additional contribution, we provide two practical explanation-derivation algorithms applicable for black-box CPSs. We start by formulating a straightforward explanation algorithm that exhaustively searches the space of potential explanations and is guaranteed to return an optimal one. While that algorithm is correct-by-construction, it is not computationally efficient. It involves searching in a high-dimensional space and running multiple system-simulation runs in perturbed, contingent environments; this process is intractable in complex scenarios. Thus, we follow with a computationally-efficient, approximated explanation algorithm, based on a causal-responsibility -based heuristic search. While slightly sacrificing optimality in some cases, its complexity scales more moderately with the problem size. We conclude the paper with experimental evaluation of our contributions for an autonomous car operating on an obstructed track. We use this system as a running example throughout this paper, for a grounded and lucid discussion. The paper is structured as follows: Section II provides preliminaries on the CPS testing model and actual causality, introduces the running example, and formulates our objectives. Section III provides the theoretical contribution: extension of the actual causality framework for CPSs. Section IV provides the practical contribution: the failure explanation algorithms. Section V provides the experimental evaluation of our algorithms for the autonomous car. Section VI concludes the paper. I-C Related work Attempts to define causality go back to Aristotle, with the modern view dating back to Hume [ 16 ] . A more recent direction is but-for causality , introduced by Lewis [ 18 ] , saying that A is a cause of B if both happened, and if A had not happened, then B would not have happened . Informally, actual causality extends and generalizes but-for causality to capture the cases of preemption and over-determination . Preemption is illustrated by the classic example of Lewis, where two children, Suzy and Billy, are throwing rocks at a bottle. Suzy’s rock hits the bottle first, shattering it. Had Suzy not thrown her rock, Billy’s rock would’ve shattered the bottle. Actual causality captures the fact that it is Suzy’s rock that is a cause of bottle shattering, but not Billy’s. Over-determination can be illustrated by an example of two arsonists dropping