From Program Slices to Causal Clarity: Evaluating Faithful, Actionable LLM-Generated Failure Explanations via Context Partitioning and LLM-as-a-Judge DOI: nnn Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; Tue 9 - Fri 12 June 2026; Glasgow, United Kingdom The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026), Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom CCS: Software and its engineering Software testing and debugging

Julius Porbeck, Christian Medeiros Adriano, Holger Giese · 2026

Read From Program Slices to Causal Clarity: Evaluating Faithful, Actionable LLM-Generated Failure Explanations via Context Partitioning and LLM-as-a-Judge DOI: nnn Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; Tue 9 - Fri 12 June 2026; Glasgow, United Kingdom The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026), Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom CCS: Software and its engineering Software testing and debugging by Julius Porbeck, Christian Medeiros Adriano, Holger Giese (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18309.txt.

Introduction

Porbeck, Adriano, and Giese test how the context fed to a debugging model shapes its failure explanations. Explanations that are incomplete or wrong can mislead bug triage and repair, yet most work treats them as a by-product. Varying context across 93 setups on real bugs, the authors score explanations on six criteria and check machine judgments against human ratings. Rich, failure-specific artifacts improve causal and actionable quality, while oversized contexts yield vague text, and top-scoring explanations predict better repair rates.

Catalog record

CREATORJulius Porbeck, Christian Medeiros Adriano, Holger Giese
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50935
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Program Slices to Causal Clarity: Evaluating Faithful, Actionable LLM-Generated Failure Explanations via Context Partitioning and LLM-as-a-Judge DOI: nnn Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; Tue 9 - Fri 12 June 2026; Glasgow, United Kingdom The 30th International Conference on Evaluation and Assessment in Software Engineering (EASE 2026), Tue 9 - Fri 12 June 2026, Glasgow, United Kingdom CCS: Software and its engineering Software testing and debugging Julius Porbeck, Christian Medeiros Adriano, Holger Giese Abstract . Large language model (LLM)-based debugging systems can generate failure explanations, but these explanations may be incomplete or incorrect. Misleading explanations are harmful for downstream tasks (e.g., bug triage, bug fixing). We investigate how explanation quality is affected by various LLM context configurations. Existing work predominantly treats LLM-generated failure explanations as an ad hoc by-product of debugging or repair workflows, using generic prompting over undifferentiated artifacts such as code, tests, and error messages rather than targeting explanations as a first-class output with dedicated quality assessment. Consequently, existing approaches provide limited support for assessing whether these explanations capture the underlying fault–error–failure mechanism and for actionable next steps, and most techniques instead prioritize task success (e.g., patch correctness or review quality) over the explicit causal explanation quality. We systematically vary the debugging information to study how distinct context compositions affect the quality of LLM-generated failure explanations. Across 93 context configurations on real bugs and three economically viable models ( gpt-5-mini , DeepSeek-V3.2 , and Grok-4.1-fast ), we evaluate explanations with six criteria and validate the LLM-as-a-judge scores against human ratings in a user study. Our results indicate that explanation quality is causally affected by context composition. Evidence-rich, failure-specific artifacts improve causal and action-oriented quality, whereas overly large contexts tend to yield vague explanations. Higher explanation-score quartiles are associated with higher downstream repair pass rates and, for some models, with fixes that are closer to the reference minimal fixes. In contrast, low-score quartiles can even underperform the no-explanation baseline. Reproduction package is publicly available. 1 1 1 https://doi.org/10.5281/zenodo.19556994 1. Introduction Context – Software failures demand a precise understanding. While large language models (LLMs) can expedite the explanation of various types of software failures, these can still be incomplete or vague. This can mislead downstream tasks such as bug triage and bug fixing, even when these tasks are automated. Hence, a lack of explanation quality more than an interpretability issue, it becomes a validation problem for the LLM-based bug-fixing pipeline. Current Limitations – Current work predominantly treats LLM-generated failure explanations as an ad hoc by-product of code review, debugging, or repair workflows. These workflows typically rely on generic prompting over entire undifferentiated artifacts such as code, tests, error messages, or pull-requests ( 9 ; 4 ; 18 ) . In practice, current LLM-based solutions have retained the automated program repair (APR) ( 20 ; 24 ) approach by prioritizing task success (e.g., patch correctness or review quality) over the quality of the accompanying explanations. Consequently, existing approaches provide limited support for assessing whether these explanations capture the underlying fault–error–failure mechanism and for actionable next steps. Insights – Explanation quality is partly subjective, yet it must capture causal knowledge about which program elements and conditions made the failure occur and what would need to change for it to disappear. We hypothesize that the reasons for incomplete or vague explanations arise from ad hoc context compositions that ignore trade-offs between criteria that can subjective (readability, conciseness), objective (faithfulness to artifacts), or actionable (conducive to valid fixes). In addition, conditioning LLMs on long token sequences – such as undifferentiated free-text and large code dumps – is known to dilute attention ( 2 ; 15 ) . Approach – We systematically investigate how manipulating the context affects the quality , faithfulness , and actionability of LLM-generated failure explanations ( 31 ) . We vary context composition using debugging artifacts across 93 configurations on real bugs from popular open-source projects and three economically viable models ( gpt-5-mini , DeepSeek-V3.2 , and Grok-4.1-fast ), assess explanation quality along six criteria, and validate LLM-as-a-judge ratings against human ratings in a user study. This paper builds on and extends the first author’s Master’s thesis ( 23 ) . Results – The strongest gains come from executable, failure-specific artifacts such as CODE and TEST ; in contrast, adding broad or prose-heavy context can make explanations vaguer. Higher explanation-score quartiles also show higher downstream repair pass rates and, for some models, fixes closer to the reference minimal fixes, while low explanation-score quartiles can underperform with respect to the no-explanation baseline. Contributions are threefold: (1) A context-module catalog and context-partitioning pipeline. We define context as composable modules and implement a pipeline that generates and evaluates explanations and downstream fixes under varied context compositions. (2) A failure-explanation-specific evaluation framework with six quality criteria. We operationalize explanation quality as a multi-dimensional construct and validate judge-based criteria against human judgments in a user study. (3) An empirical study of context partitioning and downstream repair precision. We quantify how context composition shapes explanation quality and relate explanation quality to downstream repair success and repair precision (similarity to a reference minimal fix). 2. Foundations and Related Work The automation of failure explanations overlaps with dependability theory, cognitive debugging, program analysis, and, more recently, LLM context management. We review how these areas define fault–error–failure chains, how developers actually search for causes, how program slicing and context partitioning can isolate failure-relevant evidence, and how existing evaluation work falls short of a failure-explanation–specific framework. Two trends stand out: (1) context work is shifting from “more is better” toward selective, multi-source workflows for LLM-based software engineering, and (2) evaluation work increasingly treats developer-facing text as a multi-dimensional object with distinct criteria for usefulness, grounding, and surface factors (readability and brevity). However, these trends are rarely unified for failure explanations, leaving a gap of a dedicated quality framework paired with a principled method for composing context. 2.1. Explanation Failure Role in Debugging A failure explanation answers “Why did this failure happen?” by linking an observed failure (e.g., a failing test) to the program elements and conditions that caused it, including relevant inputs and configurations ( 19 ; 30 ) . It should go beyond symptoms by identifying what must change for the failure to disappear, recognizing that a fault becomes an observable failure only if it is executed, produces an incorrect state, and that state propagates to the output ( 3 ; 28 ) . Because developers rely on multiple sources—code, inputs, and environment configurations—explanations often arise from ad hoc mental models that can be technically correct yet unhelpful for downstream tasks ( 12 ; 14 ) . Existing work has begun to add natural-language explanations to fault localization, but with different goals and granularity than ours. Yan et al. ( 32 ) combines static analysis and LLMs to generate explainable, fault-localization reports that explain crashes and evaluates them primarily through user satisfaction, without a multi-dimensional explanation-quality framework or explicit fault–error–failure chains. Widyasari et al. ( 30 ) and Salmon et al. ( 26 ) study LLM-generated explanations using criteria such as usefulness, correctness, and verbosity, but focus on step-by-step reasoning or novice error feedback for single-artifact programs rather than system-level, multi-source failure explanations.

Read the complete text