EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models

Mahdi Dhaini, Kafaite Zahra Hussain, Efstratios Zaradoukas, Gjergji Kasneci · 2025

Read EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models by Mahdi Dhaini, Kafaite Zahra Hussain, Efstratios Zaradoukas, Gjergji Kasneci (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01238.txt.

Introduction

Mahdi Dhaini and co-authors introduce a 2025 framework for testing explanation methods for language models. As such models enter high-stakes use, users need to know why a model gave an answer. Many methods claim to explain a model, and stakeholders need a fair way to compare them. The framework, EvalxNLP, is written in Python and works with transformer models.

Eight known explanation methods are built in. Users can score each by how faithful, plausible, and simple its explanations are. A language model also writes plain-text summaries of results for the user. Human testers rated the framework highly in the study. The team aims to open explanation tools to more users and to drive steady progress.

Catalog record

CREATORMahdi Dhaini, Kafaite Zahra Hussain, Efstratios Zaradoukas, Gjergji Kasneci
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01238

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EvalxNLP: A Framework for Benchmarking Post-Hoc Explainability Methods on NLP Models Mahdi Dhaini, Kafaite Zahra Hussain, Efstratios Zaradoukas, Gjergji Kasneci Abstract As Natural Language Processing (NLP) models continue to evolve and become integral to high-stakes applications, ensuring their interpretability remains a critical challenge. Given the growing variety of explainability methods and diverse stakeholder requirements, frameworks that help stakeholders select appropriate explanations tailored to their specific use cases are increasingly important. To address this need, we introduce EvalxNLP , a Python framework for benchmarking state-of-the-art feature attribution methods for transformer-based NLP models. EvalxNLP integrates eight widely recognized explainability techniques from the Explainable AI (XAI) literature, enabling users to generate and evaluate explanations based on key properties such as faithfulness, plausibility, and complexity. Our framework also provides interactive, LLM-based textual explanations, facilitating user understanding of the generated explanations and evaluation outcomes. Human evaluation results indicate high user satisfaction with EvalxNLP , suggesting it is a promising framework for benchmarking explanation methods across diverse user groups. By offering a user-friendly and extensible platform, EvalxNLP aims at democratizing explainability tools and supporting the systematic comparison and advancement of XAI techniques in NLP. [[email protected], ] *1 [[email protected], ] [[email protected], ] [[email protected], ] keywords Natural Language Processing ,Explainable AI ,Feature Attribution ,Evaluation ,Large Language Models † † address: Technical University of Munich, School of Computation, Information and Technology, Department of Computer Science, Boltzmannstr. 3, Garching, 85748, Germany † † corresp: Corresponding author. 1 Introduction Although significant progress has been made in the field of NLP, transformer-based models often operate as black boxes , making it challenging to interpret their decision-making processes. In high-stakes domains like medical diagnosis and financial decision-making, transparency is essential for trust and accountability. Despite the rapid development of explainability methods, there remains a lack of standardized evaluation frameworks, particularly for NLP, where text data is inherently unstructured and context-dependent. Existing explainability assessments vary widely, spanning qualitative user studies and quantitative metrics like faithfulness and plausibility, yet no universal consensus exists on the most effective approach. To address these challenges, we introduce EvalxNLP , a benchmarking framework for evaluating post-hoc explainability methods in text classification tasks. EvalxNLP supports multiple explanation techniques, and assesses them across key properties such as faithfulness, plausibility, and complexity. In addition, EvalxNLP integrates LLM-based natural language explanations to facilitate the users’ understanding of the generated explanations and evaluations. We also conduct a user-based study to evaluate the usability and user satisfaction with the framework. Our framework provides a systematic, user-friendly platform that aims to democratize access to explainability tools, enabling both researchers and practitioners to compare and refine XAI techniques for NLP applications. By offering a unified and reproducible evaluation methodology, EvalxNLP advances the field of explainability, promoting more transparent and trustworthy AI systems. 2 Related Work Most existing explainability evaluation frameworks are designed for general-purpose applications, meaning that they include explainability methods used in image or tabular applications. As a result, most of them lack dedicated support for text-based models. OpenXAI Agarwal et al. 2022 , BEExAI Sithakoul et al. 2024 , and Quantus Hedström et al. 2023 are prime examples of supporting multiple data modalities but not including text-specific evaluation metrics. Performance assessment is typically based on generic criteria, without tailored adaptations for NLP tasks. Frameworks such as Inseq Sarti et al. 2023 support sequence generation models but lack built-in evaluation metrics. XAI-Bench Liu et al. 2021 evaluates explainability methods using synthetic data, which may not fully capture real-world text applications Faber et al. 2021 . While these frameworks provide partial solutions, they do not offer a comprehensive suite for text explainability. ferret Attanasio et al. 2023 facilitates explainability evaluation for NLP models but supports only five feature-attribution methods and six evaluation metrics. Among existing XAI libraries, it is the only one with adequate text-specific explainability features, to be considered as an NLP-specialized explainability framework. But, it relies on some metrics that have been shown to be inaccurate especially for measuring faithfulness (as explained in section 3.3.1 ); Captum Miglani et al. 2023 provides implementations for 22 attribution methods but is limited to two evaluation metrics (Infidelity and Sensitivity) and lacks built-in benchmarking. Similarly, AIX360 Arya et al. 2022 supports text data but evaluates only two properties, faithfulness and monotonicity, without systematic benchmarking capabilities. M4 Li et al. 2023 evaluates faithfulness for image and text modalities but does not assess plausibility or complexity. While these frameworks address various aspects of explainability, among existing XAI libraries, only ferret offers a complete set of key features Attanasio et al. 2023 : multiple explainability methods, Transformers-readiness (built with close integration into the Hugging Face (HF) transformers library), evaluation APIs, explainable datasets (i.e., those with human-annotated rationales), and built-in visualization. In addition to these features, our tool also extends functionality by incorporating recent explainability methods, recent metrics for evaluating explanation properties and providing an LLM-based module that generates natural language explanations to enhance user understanding. It consolidates capabilities scattered across multiple frameworks, offering a robust suite for benchmarking, evaluation, and qualitative explanations. By seamlessly integrating these features into one comprehensive framework, like EvalxNLP , we support practitioners in benchmarking Ph-FA explanation methods. 3 The Framework The EvalxNLP framework builds on the top of four main components described below. For the technical implementation details, we refer the reader to the documentation provided in the repository. We release the code, tutorials, and documentation in the following repository 1 1 1 https://github.com/dmah10/EvalxNLP . Table 1 : Overview of the built-in explanation methods, metrics, and datasets in EvalxNLP Method Category Explainers Property Metric Dataset Classification Task Gradient-based Saliency Faithfulness Soft suff MovieReviews Sentiment GradientxInput Softcomp HateXplain Hate Speech IntegratedGradients FAD N-AUC e-SNLI NLI DeepLIFT AUC-TP Perturbation-based LIME Plausibility IOU-F1 Guided BackProp Token-F1 SHAPIQ AUPRC SHAP Complexity Complexity Sparseness 3.1 Explainers One goal of EvalxNLP is to enable users to generate diverse explanations through multiple explainability methods. The explainer component integrates eight widely recognized explainability methods from the XAI literature, specifically focusing on post-hoc feature attribution (Ph-FA) methods. EvalxNLP incorporates two categories of post-hoc methods: gradient-based and perturbation-based approaches. Gradient-based methods compute feature importance by leveraging gradients of the model’s output with respect to its input features. They efficiently utilize backpropagation, making them well-suited for deep learning models. EvalxNLP integrates five key methods, including Saliency (also called Gradients) Simonyan et al. 2013 , which calculates raw gradients to highlight important inputs; Gradient×Input Shrikumar et al. 2017 , which scales gradients by input values for enhanced clarity; Integrated Gradients Sundararajan et al. 2017 , which averages gradients along a path from a baseline to the input; DeepLift Shrikumar et al. 2017 , which attributes differences in activations to individual inputs for more stable attributions; and Guided BackProp Springenberg et al. 2014 , which filters negative gradients to highlight only positive contributions. EvalxNLP implements these methods using Captum and ferret, providing a diverse set of efficient and interpretable attribution techniques.. Perturbation-based methods integrated into the toolbox include the widely used LIME Ribeiro et al. 2016 and SHAP Lundberg and Lee 2017 methods, as well as the recently introduced SHAP with interactions method (SHAP-I) Muschalik et al. 2024 , which augments traditional Shapley values by incorporating feature interactions Fumagalli et al. 2023 , a notable contribution over existing frameworks.

Read the complete text