Counterfactual Simulatability of LLM Explanations for Generation Tasks
Marvin Limpijankit, Yanda Chen, Melanie Subbiah · 2025
Read Counterfactual Simulatability of LLM Explanations for Generation Tasks by Marvin Limpijankit, Yanda Chen, Melanie Subbiah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21740.txt.
Introduction
Marvin Limpijankit, Yanda Chen, Melanie Subbiah, Nicholas Deas, and Kathleen McKeown test whether AI explanations actually explain in this two thousand twenty-five paper. Language models are unpredictable: small prompt changes can swing outputs. A good explanation should let a user predict what the model would do on related counterfactual inputs, a property called counterfactual simulatability. It had been studied only for yes-or-no questions. The authors extend the evaluation framework to generation tasks.
Two case studies anchor the work: news summarization and medical suggestion. In summarization, explanations did help users predict the model's counterfactual outputs. In medical suggestion, there was significant room for improvement. The pattern suggests simulatability evaluation suits skill-based tasks better than knowledge-based ones, where the model's hidden information, not its reasoning, drives outputs. The framework gives explanation researchers a sharper tool for high-stakes settings.
Catalog record
| CREATOR | Marvin Limpijankit, Yanda Chen, Melanie Subbiah |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21740
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Counterfactual Simulatability of LLM Explanations for Generation Tasks Marvin Limpijankit, Yanda Chen, Melanie Subbiah Abstract LLMs can be unpredictable, as even slight alterations to the prompt can cause the output to change in unexpected ways. Thus, the ability of models to accurately explain their behavior is critical, especially in high-stakes settings. Counterfactual simulatability measures how well an explanation allows users to infer the model’s output on related counterfactuals and has been previously studied for yes/no question answering. We provide a general framework for extending this method to generation tasks, using news summarization and medical suggestion as example use cases. We find that while LLM explanations do enable users to better predict their outputs on counterfactuals in the summarization setting, there is significant room for improvement for medical suggestion. Furthermore, our results suggest that evaluating counterfactual simulatability may be more appropriate for skill-based tasks as opposed to knowledge-based tasks. 1 Introduction While large language models (LLMs) have proven effective for a diverse range of applications, their outputs still often contain hallucinations of unsupported information ( Ji et al., 2023 ) or biases ( Sheng et al., 2021 ) that hinder their reliability on critical language generation tasks. At the same time, it is important that users themselves can accurately evaluate the model capabilities, knowledge, and reliability ( Steyvers et al., 2025 ) . Particularly in high stakes domains, such as medical applications, misunderstandings or the lack of ability to predict model behavior on unseen inputs can pose disastrous risks to users ( Michalowski et al., 2024 ) . To anticipate these risks, recent work has turned attention to evaluating the reliability of LLM explanations ( Madsen et al., 2024 ; Turpin et al., 2023 ; Kunz and Kuhlmann, 2024 ) . In particular, Chen et al. (2024) evaluates the counterfactual simulatability of LLM explanations for yes/no question answering. Counterfactual simulatability is a measure of how well a model’s explanation allows humans to correctly infer the model’s predictions on simulatable counterfactuals (unseen inputs where the explanation should enable the user to confidently guess the model’s output). Furthermore, according to Chen et al. (2024) , the counterfactual simulatability of an explanation can be decomposed into simulation generality , a measure of the diversity of simulatable counterfactuals and simulation precision , a measure of the proportion of these counterfactuals for which humans correctly infer the model’s output. Ideal model explanations should balance both generality and precision. Counterfactual simulatability, however, is equally critical to generation tasks, where the larger space of possible outputs makes understanding a model’s decision process more challenging. To fill this gap, we formalize a framework for evaluating counterfactual simulatability in language generation tasks (Figure 1 ). To evaluate a model’s explanation, we first use a separate LLM to (1) decompose the explanation into atomic units and (2) generate relevant counterfactuals. Then, an annotator (human or the LLM) evaluates the simulatability and precision of each unit of the model’s explanation using the counterfactual and the model’s output on the counterfactual respectively. Finally, generality and precision scores are calculated following § 4.4 . Rather than evaluating the factual correctness of explanations, our framework measures the ability of LLMs to accurately describe their behavior in a generative setting. We evaluate whether LLMs’ explanations lead to reliable human mental models that are consistent with their outputs . Figure 1: Our evaluation pipeline. Given a model’s explanation, an LLM is prompted to decompose the explanation into atomic units (left) and generate relevant counterfactuals (right). For each unit, an annotator verifies whether the element appears in the counterfactual (simulatability) and the counterfactual output (precision). We apply the framework to two tasks: news summarization using CNN/DM ( Nallapati et al., 2016 ) and medical suggestion generation using the Taiwan e-Hospital Dataset ( Chen et al., 2022b ) . These tasks involve different trade-offs between generality and precision . Explanations for news summarization can be highly general as models may employ similar approaches for documents that contain common elements (e.g., dates, quotes) but differ in content. In contrast, medical suggestion explanations may be less general as the model’s response can be highly dependent on the user input (i.e., minor variances in expressed symptoms can lead to different suggestions). This can lead to a more limited set of simulatable counterfactuals, though potentially being more precise in its outputs. With these generation tasks, we conduct an initial evaluation of LLMs’ explanations and assess where models’ explanations in these generation tasks fail. In summary, our contributions are as follows: 1. We propose a framework to measure the ability of LLMs to accurately describe their behavior in a generative setting by evaluating whether LLMs’ explanations lead to mental models consistent with their outputs (counterfactual simulatability). 2. We assess the feasibility of using an LLM to automate human annotation in our evaluation pipeline and find that doing so achieves agreement with humans comparable to agreement among human annotators. 3. Using our framework, we evaluate Chain-of-thought and Post-hoc explanations for multiple LLMs on two complementary tasks: news summarization and medical suggestion generation. We show that LLM explanations lead to reliable mental models in news summarization, but not medical suggestion. 1 1 1 Our code is available at github.com/mlimpijankit/counterfactual-simulatability-generation-tasks 2 Related Work Human mental models. Humans form mental models of the physical world as a whole ( Gentner and Stevens, 2014 ) as well as specific technologies (e.g., Payne, 1991 ; Du et al., 2018 ; Lei et al., 2016 ) through their past experiences and observations. Specifically with regard to artificial intelligence systems, explanations of model predictions have been considered high quality if they provide users with an accurate and generalizable understanding of the system in the form of mental models ( Rutjes et al., 2019 ; Merry et al., 2021 ) . When such explanations are successful, they can improve users’ ability to effectively use AI models ( Vasconcelos et al., 2022 ; Senoner et al., 2024 ) as well as to anticipate and correct undesirable model behaviors, such as biases and incorrect predictions ( Bansal et al., 2019 ) . We specifically evaluate LLMs’ explanations in generation tasks considering their ability to help form mental models. Explanation evaluation. A variety of different approaches have been used to evaluate the quality and utility of model-generated natural language explanations and rationales. Studies evaluating natural language explanations similar to word attributions ( Huang et al., 2023 ; Madsen et al., 2024 ) as well as unconstrained explanations ( Turpin et al., 2023 ) have focused on faithfulness measures. In particular, work has proposed metrics for dimensions including comprehensiveness ( DeYoung et al., 2020 ) , sufficiency ( DeYoung et al., 2020 ) , alignment with human rationales ( Fayyaz et al., 2024 ) , plausibility ( Wojciechowski et al., 2024 ) , and scrutability ( Xu et al., 2023 ) among others. In contrast to intrinsic measures of model explanations, other work evaluates explanations extrinsically through impact on task performance (e.g., Camburu et al., 2018 ; Wei et al., 2022 ; Krishna et al., 2023 ). Among these topics, relatively few works have investigated natural language explanations in language generation tasks; such studies include dialogue responses ( Zhou et al., 2021 ) , dialogue understanding (