Prompt-Counterfactual Explanations for Generative AI System Behavior
Sofie Goethals, Foster Provost, João Sedoc · 2026
Read Prompt-Counterfactual Explanations for Generative AI System Behavior by Sofie Goethals, Foster Provost, João Sedoc (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03156.txt.
Introduction
Sofie Goethals, Foster Provost, and João Sedoc ask a pointed question about generative AI. What is it about a prompt that makes the system produce output with a specific trait, such as toxicity, negative sentiment, or political bias? Groups deploying these systems need causal answers of exactly that kind, more and more. The authors adapt counterfactual explanations, a standard explainable-AI technique, to this setting.
Off-the-shelf counterfactuals fail here, because generative systems are non-deterministic and open-ended. The proposed framework works where downstream classifiers can reveal key traits of the outputs. An algorithm built on it produces prompt-counterfactual explanations: minimal prompt changes that would flip the trait. The authors demonstrate the approach on generative tasks. It gives decision-makers a concrete way to trace system behavior back to input choices.
Catalog record
| CREATOR | Sofie Goethals, Foster Provost, João Sedoc |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prompt-Counterfactual Explanations for Generative AI System Behavior Sofie Goethals, Foster Provost, João Sedoc Abstract As generative AI systems become integrated into real-world applications, organizations increasingly need to be able to understand and interpret their behavior. This paper examines a key question: what is it about the input—the prompt—that causes an LLM-based generative AI system to produce output that exhibits specific characteristics. We adapt a common technique from the Explainable AI literature: counterfactual explanations. We explain why traditional counterfactual explanations cannot be applied directly to generative AI systems, due to four challenges, and then propose a flexible framework that adapts counterfactual explanations to non-deterministic, generative AI systems in scenarios where downstream classifiers can reveal key characteristics of their outputs. Based on this framework, we introduce an algorithm for generating prompt-counterfactual explanations (PCEs). We demonstrate the production of counterfactual explanations with three case studies, exhibiting different output characteristics (viz., political leaning, toxicity, and sentiment) and different characteristic distributions. The case studies demonstrate the production of PCEs, addressing the four challenges, and suggest that PCEs could streamline prompt engineering to suppress undesirable output characteristics and enhance red-teaming efforts. Ultimately, this work lays a foundation for prompt-focused interpretability in generative AI, a necessary capability as these models are applied to higher-stakes tasks with requirements for transparency and accountability. Prompt-Counterfactual Explanations Keywords: Explainable AI, Generative AI, LLMs, Counterfactual Explanations, Transparency 1 Introduction As firms work to integrate large-scale generative AI systems across functions, they increasingly need to be able to understand not just what the systems generate, but also why they generate it. Generative AI systems are now being used in sales, marketing, operations, research, recommender systems, healthcare, education, and many more functions ( 53 ; 35 ; 59 ; 27 ; 21 ) . However, generative AI system outputs are complex, typically unstructured, and often non-deterministic, which makes them quite challenging to monitor, evaluate, analyze, and explain. Generative AI refers to a broad class of systems that produce text, images, audio, or other content based on learned distributions ( 20 ) . In this paper, we will focus on AI systems based on large language models (LLMs), which currently represent the most widely applied subclass of generative AI systems. 1 1 1 We use publicly available models from the HuggingFace model library https://huggingface.co/models for our experiments. As of Dec 7th, 2025, 300,930 out of 2,263,421 HuggingFace models that text generation models. Despite our specific focus on LLMs, this methodology can be applied to other generative modalities. As we will describe in more detail below, one main challenge of explaining the behavior of generative AI systems is that system outputs are very complex. Even in the simplest cases of text generation, the output of an LLM-based system is a textual response—essentially an entire document. What would it even mean to examine “the behavior” of such a system? In this paper, we propose a framework for adapting and applying counterfactual (CF) explanations ( 43 ; 64 ; 18 ) to generative AI systems. Counterfactual explanations, a popular method for explaining the behavior of traditional predictive systems, identify (minimal) changes to the input that would alter the AI system’s prediction, decision, or action. However, traditional CF explanations cannot be applied directly to generative AI systems for several reasons, requiring rethinking and extending the notion of CF explanations. Except in very restrictive settings, generative AI systems do not produce discrete predictions, decisions, or actions (hereafter, decisions). Instead, they generate complex outputs; for example, LLMs produce open-ended text. In addition, the outputs of the generative AI system are very often stochastic. Even with an identical input prompt, the outputs can vary from run to run. For this paper, we focus on explaining a specific class of AI system behaviors: whether the output exhibits a certain characteristic (often undesirable). Behaviors in this class include things like: did the system produce hate speech? Is something in the system’s output fabricated? Does the system’s output violate company policy or otherwise require content moderation action? Is the system’s output biased in a certain way? Does the system’s output exhibit very negative sentiment ( 32 ; 1 ; 52 ; 34 ) ? Other applications may require the identification of task-specific characteristics. For example, in personalized recommender systems, a firm may want to evaluate whether the generated suggestions align with a user’s preferences and therefore measure elements such as diversity, novelty and bias in the output ( 12 ) . For a deployed chat system, we may want to assess whether it exhibits traits analogous to empathy, or analyze its generated conversation with respect to the Big Five personality traits ( 14 ; 40 ; 57 ) . We will refer to this sort of behavior characteristic as downstream classification ; ideally such characteristics can be estimated well via classifiers downstream from the generated output. 2 2 2 For this paper, we assume that such a sufficient classifier exists; whether a particular generated output characteristic truly can be estimated well will depend on the characteristic. This paper focuses on explaining what about the input leads generative AI outputs to exhibit a particular behavior as revealed by downstream classification. As we shall discuss in detail, there are many different uses of the term “counterfactual explanation.” This paper focuses specifically on explaining what it is about the input to the generative model—the prompt—that leads the output to exhibit the focal characteristic. Therefore, for clarity we will call these explanations “prompt-counterfactual explanations” or PCEs. In Section 3 , we discuss four key challenges to applying CF explanations to generative AI systems, and outline how the CF explanation framework can be adapted to address these challenges and allow the production of PCEs for generative AI systems. Specifically, we provide an answer to the question: “How can we define and compute a meaningful counterfactual explanation in the context of a generative AI pipeline? ” The key contributions of this paper are as follows. To our knowledge, this is the first systematic adaptation of the standard notion of counterfactual explanations to generative AI systems with downstream classifiers. 3 3 3 The term “counterfactual” has many uses in AI, including in other sorts of explanations that do not correspond to the traditional notion of counterfactual explanations. • We present and explain the challenges of applying traditional CF algorithms to a generative AI system. • We present a PCE solution that addresses all these challenges for the downstream-classification setting. • We present case studies demonstrating the production of PCEs across three different LLM use cases, showing their utility and illustrating different facets of the solution. The rest of the paper is structured as follows. In Section 2 , we contextualize our contribution with prior work on Explainable AI in general and specifically for generative AI systems. We present the problem statement and the setup for generative AI PCEs in Section 3 . Here, we discuss how to adapt counterfactual explanations to deal with generative AI’s non-determinism, by focusing on aggregate properties of the model outputs. In Section 4 , we present a straightforward, first PCE algorithm for the input/output behavior of the generative AI system. Section 5 presents results across three illustrative case studies, highlighting scenarios where PCEs could be useful. Specifically,