CafGa: Customizing Feature Attributions to Explain Language Models
Alan Boyle, Furui ChengVilém ZouharMennatallah El-AssadyETH Zurich · 2025
Read CafGa: Customizing Feature Attributions to Explain Language Models by Alan Boyle, Furui ChengVilém ZouharMennatallah El-AssadyETH Zurich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20901.txt.
Introduction
Explaining a language model's answer usually means scoring each word's influence, which is slow and often meaningless. Alan Boyle and three co-authors present CafGa, an interactive tool that fixes both problems, in a 2025 paper for the EMNLP 2025 demo track. CafGa lets users explain at customizable levels of detail.
Instead of single words, users can define larger text segments as the units of explanation. The tool computes feature attributions over those segments and draws deletion and insertion curves to judge explanation quality. Interactive segmentation keeps long documents tractable. Semantic units, such as phrases or sentences, carry meaning that single words cannot. The authors position CafGa as a workbench for researchers who need explanations they can adapt and evaluate.
Catalog record
| CREATOR | Alan Boyle, Furui ChengVilém ZouharMennatallah El-AssadyETH Zurich |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20901
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CafGa: Customizing Feature Attributions to Explain Language Models Alan Boyle, Furui ChengVilém ZouharMennatallah El-AssadyETH Zurich Abstract Feature attribution methods, such as SHAP and LIME, explain machine learning model predictions by quantifying the influence of each input component. When applying feature attributions to explain language models, a basic question is defining the interpretable components. Traditional feature attribution methods, commonly treat individual words as atomic units. This is highly computationally inefficient for long-form text and fails to capture semantic information that spans multiple words. To address this, we present CafGa, an interactive tool for generating and evaluating feature attribution explanations at customizable granularities. CafGa supports customized segmentation with user interaction and visualizes the deletion and insertion curves for explanation assessments. Through a user study involving participants of various expertise, we confirm CafGa’s usefulness, particularly among LLM practitioners. Explanations created using CafGa were also perceived as more useful compared to those generated by two fully automatic baseline methods: PartitionSHAP and MExGen, suggesting the effectiveness of the system. 1 1 footnotetext: Corresponding author 2 2 footnotetext: CafGa is available as a pip-installable package cafga . The source code is hosted on https://github.com/explain-llm/CafGa , and a live demo is accessible at cafga.ivia.ch . 1 Introduction Large Language Models (LLMs) continuously evolve in scale and capabilities across a wide range of tasks, such as question answering, reasoning, and text summarization ( Kamalloo et al. 2023 , inter alia) . Meanwhile, their increasing complexity poses significant challenges in interpretability and human trust ( Sun et al. 2024 ) . Feature attribution methods, e.g. SHAP ( Lundberg and Lee 2017 ) and LIME ( Ribeiro et al. 2016 ) , offer a promising approach to explain LLM behaviors by quantifying the influences of each component in the input. These explanations calibrate users’ trust in model predictions ( Bansal et al. 2021 ) and offer useful insights into the prediction shortcuts and model biases ( Du et al. 2021 ; Ren and Xiong 2023 ) . Figure 1: Feature attribution explanations quantify the contributions of each component (e.g., words), allowing users to validate the models’ reasoning. Defining interpretable components is a fundamental question in generating meaningful feature attributions ( Ribeiro et al. 2016 ) . Traditional removal-based methods typically treat individual words as the basic units of interpretation ( Figure 1 ), but this approach struggles to scale effectively for long-form text due to computational inefficiency and the fact that meaningful semantic cues often span multiple tokens or phrases, such as “made it worth it” in Figure 1 . To address this, we propose CafGa, an interactive tool that enables users to create feature attribution explanations at customized levels of granularity. The tool offers default options to segment text at the word, sentence, or paragraph level, and supports users to further refine these segments by interactively isolating text spans. CafGa also provides functionality to evaluate and compare different segmentation strategies. The system visualizes deletion and insertion curves, showing how the model’s prediction changes as components are removed or added based on their attribution rankings, and calculates the area under the perturbation curve as the fidelity scores. We conducted a two-stage user study to evaluate the usability and effectiveness of CafGa. In the first phase, ten participants used CafGa to create explanations that helped them understand the model’s reasoning. Based on self-reported usability scores, users with machine learning expertise generally found the system easy to use and learn, while novice users experienced some difficulty due to the need to grasp new concepts before effectively interacting with the tool. Despite this, participants across all expertise levels reported that CafGa was helpful in improving their understanding of the LLM. In the second phase, we invited four expert users to compare the usefulness of the CafGa-generated explanations from the first phase with those produced by two automated baselines: PartitionSHAP ( Lundberg 2024 ) and MExGen ( Paes et al. 2024 ) . In 64% of the comparisons, participants rated the human-generated explanations as the most helpful, demonstrating the effectiveness of the proposed system in supporting meaningful model interpretation. Figure 2: The user creates a task where a model is asked to decide whether to recommend a restaurant to a customer. 2 System Design CafGa is an interactive system that includes a task creation page for defining prediction tasks and customizing explanation granularities, and an explanation page that displays explanations and their evaluation results. The following sections introduce the key features and design. Figure 3: An overview of CafGa: The user first creates a predictive task (A) and then decides the granularities for explanations by assigning words into groups and gets an explanation based on the assignments (B). Using the perturbation curve the user can validate the fidelity of the generated explanation (C). Define a Prediction Task. CafGa allows users to define a prediction task by constructing a prompt and defining an evaluator ( Figure 2 ). To construct the prompt, users need to complete the template field with the structure of the prompt and the input field with the content they wish to explain. We use the design in ChainForge ( Arawjo et al. 2024 ) where an evaluator is used to structure the text generated by the model by transforming the text into a boolean value based on user-defined rules. To define the evaluator, users can choose from predefined operators such as Contains , which returns a boolean value indicating whether the text includes the specified Target Answer , and Entails , which assesses whether the text logically entails the Target Answer . See the full list of operators in Appendix B . Figure 4: CafGa allows users to interactively segment text to customize the explanation granularity. Customize Text Segmentation. CafGa supports default options for creating text segments, such as dividing text by word, sentence, or paragraph. Users can customize the segmentation through interactions. For example, they could brush to select a phrase and isolate the selected phrase from the sentence ( Figure 4 ). The segments may not overlap to avoid ambiguous attributions. Calculate and View Feature Attributions. CafGa uses the KernelSHAP Lundberg and Lee 2017 algorithm to sample and compute the feature attributions. The system generates random samples by removing text segments in the user-specified granularities. For each perturbed sample, the system requests ten responses, which are converted into Boolean values using the user-defined evaluator. The estimated probability, e.g., P(the answer contains ‘‘no’’) is computed as the proportion of responses evaluated as true. Finally, the system uses a weighted linear regression model to estimate the Shapley values of each group. This step runs in the back-end and is hidden from users. After the calculation, we visualize the explanations using a heatmap with a color-blind friendly color schema ( Figure 3 B) accompanied by the task description ( Figure 3 A). Evaluate Explanations. To ensure that the created explanations accurately reflect the model’s decision making and mitigate users’ confirmation bias, we adapt a commonly used and general fidelity metric, Area Over the Perturbation Curve ( Samek et al. 2017 ) . Following Petsiuk et al. 2018 , we employ two variants of the perturbation curve: Deletion and Insertion . Deletion begins with all the features present and then iteratively removes the