CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures
Akash Bonagiri, Devang Borkar, Gerard Janno Anderias, Houman Homayoun · 2026
Read CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures by Akash Bonagiri, Devang Borkar, Gerard Janno Anderias, Houman Homayoun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25338.txt.
Introduction
Akash Bonagiri and colleagues turn failed agent runs into lessons instead of just logging or retrying them. Their CausalFlow framework treats each failed trace as a chain of dependent steps, scoring steps by testing what changes when one is altered. Steps that cause failure get edits that flip the outcome to success, forming wrong-and-corrected step pairs. Those pairs support quick test-time repair with little drift, and training supervision for later improvement. Across four benchmarks in math, coding, question answering, and medical browsing, causal attribution beats heuristic fixes.
Catalog record
| CREATOR | Akash Bonagiri, Devang Borkar, Gerard Janno Anderias, Houman Homayoun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86040 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.25338
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures Akash Bonagiri, Devang Borkar, Gerard Janno Anderias, Houman Homayoun Abstract Large language model (LLM) agents frequently fail on multi-step tasks involving reasoning, tool use, and environment interaction. While such failures are typically logged or retried heuristically, they contain structured signals about where execution broke down. We introduce CausalFlow , an interventional framework that converts failed agent traces into minimal counterfactual repairs and reusable supervision. CausalFlow models execution traces as sequential chains of dependent steps and computes Causal Responsibility Scores (CRS) via step-level counterfactual intervention to identify failure-inducing steps. For these steps, we generate minimally edited repairs that flip the final outcome to success, producing validated contrastive pairs of the form (wrong step, corrected step). CausalFlow supports two complementary uses: targeted test-time repair that recovers from failures with minimal behavioral drift, and training-time supervision suitable for offline preference optimization or reward modeling. Across four benchmarks spanning mathematical reasoning, code generation, question answering, and medical browsing, CausalFlow converts failed executions into validated minimal repairs with high minimality and causal-consensus scores, and demonstrates that causal attribution is necessary for reliable improvement across diverse agent tasks, outperforming heuristic refinement in complex retrieval settings while producing more localized repairs throughout. These results demonstrate that interventional analysis over structured execution traces provides a principled and scalable mechanism for transforming agent failures into reliability gains and learning-ready supervision. 1 Introduction Large language model (LLM) agents execute complex multi-step procedures involving reasoning, tool invocation, and environment interaction ( 20 ; 40 ; 13 ; 33 ; 36 ) . When such executions fail, supervision is typically restricted to outcome-level feedback (e.g., final answer correctness), leaving ambiguous which intermediate decision caused the failure ( 19 ) . This ambiguity limits both reliability and learning: without principled credit assignment over execution traces, failures cannot be systematically repaired or converted into structured supervision. Prior approaches address agent failures through heuristic retries, critique-driven rewriting, or full solution regeneration ( 31 ; 22 ; 35 ; 12 ; 8 ; 34 ) . While effective in some cases, these strategies do not explicitly test causal responsibility within structured traces. As a result, they neither isolate the failure-inducing step nor guarantee that repairs correspond to minimal counterfactual corrections. We introduce CausalFlow , a framework for interventional analysis and counterfactual repair of multi-step agent executions. CausalFlow models an execution trace as a sequential chain of dependent steps, where each step’s output propagates forward to downstream computations. Given a failed trace, it performs step-level counterfactual interventions: replacing a candidate step and re-executing downstream computation to test whether the final outcome changes. This defines a Causal Responsibility Score (CRS) , identifying steps whose intervention flips failure to success. For causally responsible steps, CausalFlow generates minimally edited counterfactual repairs and validates them via deterministic re-execution or outcome prediction. Each validated repair yields a structured contrastive pair (wrong step, corrected step), transforming execution failures into reusable supervision ( Figure 1 ). This enables deploy-time repair that recovers from failures without parameter updates. We evaluate CausalFlow across four benchmarks spanning mathematical reasoning (GSM8K), code generation (MBPP), question answering (SealQA Hard), and medical browsing (MedBrowseComp), totaling over 3,000 problems. CausalFlow converts 42.7% of failed executions into validated minimal repairs and demonstrates measurable improvements in downstream task success through targeted counterfactual repair. Our contributions are: 1. CausalFlow : an interventional framework for step-level causal attribution in structured agent traces. Given a failed execution, CausalFlow replaces candidate steps and re-executes downstream computation to identify decisions whose intervention flips the final outcome. 2. Counterfactual repair : a minimal repair mechanism that converts causally responsible failures into validated supervision pairs (s_{i},s_{i}^{\star}) . These pairs support deploy-time repair and provide learning-ready contrastive examples for offline preference optimization or reward modeling. 3. Empirical validation : evaluation across four domains, including mathematical reasoning, code generation, question answering, and medical browsing. Across more than 3,000 problems, CausalFlow converts 42.7% of failed executions into validated minimal repairs and improves downstream task success through targeted intervention. 4. Failure-to-supervision formulation : a general formulation for transforming logged execution failures into reusable step-level supervision, enabling offline learning from naturally occurring agent failures without requiring manual annotation of every intermediate step. Failed Trace \tau=(s_{1},\ldots,s_{n}) Causal Step Identification Intervene on each step s_{i} Compute CRS (s_{i})=\mathbb{I}[\mathcal{V}(\tau^{\prime})=1] Counterfactual Repair For each s_{i} with CRS =1 Generate repairs s^{\prime}_{i} , rank by minimality Multi-Agent Validation Re-execute \tau[i\leftarrow s^{\prime}_{i}] Keep if \mathcal{V}(\tau^{\prime})=1 Validated Repairs (s_{i},s^{\prime}_{i},\tau) contrastive pairs Figure 1: Overview of CausalFlow pipeline for causal attribution and counterfactual repair. Given failed execution trace, we identify causally responsible steps via interventional scoring (CRS), generate minimal counterfactual repairs & validate repairs through re-execution or outcome prediction. 2 Related Work Iterative refinement and self-repair. A large body of recent work improves multi-step LLM agents through iterative refinement, self-reflection, and tool-assisted repair mechanisms. Reflexion introduces verbal reinforcement learning where agents critique prior outputs and store reflective feedback for future attempts ( 31 ) . Self-Refine proposes iterative generation with model-produced feedback to improve responses without additional supervision ( 22 ) . Self-Reflection generates structured error diagnoses to guide re-answering ( 29 ) . ReAct interleaves reasoning traces with tool calls to enable interactive problem solving ( 36 ) , while Tree-of-Thoughts expands reasoning through structured search over intermediate steps ( 35 ) , CRITIC incorporates external tools to validate and revise model outputs ( 12 ) , and recent self-debugging approaches enable models to iteratively repair generated code using execution feedback ( 8 ) . Agentic systems such as SWE-agent further scale iterative debugging to large software repositories ( 34 ) . Although these methods improve reliability through critique loops, search, or execution feedback, they generally rely on heuristic refinement or full regeneration rather than explicitly identifying and validating the causally responsible step via counterfactual intervention. Step-level supervision and preference learning. Alignment methods such as RLHF train reward models from human preference comparisons and optimize policies using reinforcement learning ( 26 ; 16 ) , and recent preference optimization frameworks such as Direct Preference Optimization (DPO) ( 28 ) , Ranking-based Reward Learning (RRHF) ( 37 ) , Odds-Ratio Preference Optimization (ORPO) ( 15 ) , and offline RL approaches such as Implicit Language Q-Learning (ILQL) ( 32 ) aim to improve credit assignment and stability without full RL pipelines. Nonetheless, preference signals are typically defined at the trajectory or response level, leaving ambiguity about which intermediate step caused failure. Failure diagnosis and causal intervention. MAST ( 5 ) taxonomizes 14 failure modes but neither intervenes nor repairs. Who&When ( 39 ) benchmarks failure attribution and finds state-of-the-art models achieve only 14.2% step-level accuracy from logs alone. DoVer ( 21 ) is the most closely related concurrent work, but targets a structurally distinct setting: multi-agent systems with explicit orchestrator/sub-agent topology. DoVer requires framework-level checkpoint and replay infrastructure, segments traces into planning-execution trials using re-plan steps as cut points, and first generates log-based attribution hypotheses before validating them through targeted intervention. CausalFlow differs along three axes: it targets single-agent sequential traces without requiring framework modification or checkpointing infrastructure; its Causal Responsibility Score is a purely interventional quantity computed by directly replacing candidate steps and propagating effects forward through affected descendants, bypassing the intermediate