ExplainBench: A Benchmark Framework for Local Model Explanations in Fairness-Critical Applications
James Afful · 2025
Read ExplainBench: A Benchmark Framework for Local Model Explanations in Fairness-Critical Applications by James Afful (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06330.txt.
Introduction
James Afful built ExplainBench, a shared test bench for tools that explain single machine learning decisions, in a 2025 paper. Such tools matter where models shape court, loan, and health choices, yet each tool is usually judged on its own terms, so fair comparison is hard.
ExplainBench wraps known explainers such as SHAP and LIME in one suite with full training and scoring pipelines. It grades each method on fidelity, sparsity, and robustness, and adds a web interface for hands-on study. Afful shows the bench on fairness data sets like COMPAS and lending records. Teams can now check whether an explanation holds up before trusting it in high-stakes work.
Catalog record
| CREATOR | James Afful |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06330
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ExplainBench: A Benchmark Framework for Local Model Explanations in Fairness-Critical Applications James Afful Abstract As machine learning systems are increasingly deployed in high-stakes domains—such as criminal justice, finance, and healthcare—the demand for interpretable and trustworthy models has intensified. Despite the proliferation of local explanation techniques, including SHAP, LIME, and counterfactual methods, there exists no standardized, reproducible framework for their comparative evaluation, particularly in fairness-sensitive settings. We introduce ExplainBench , an open-source benchmarking suite for systematic evaluation of local model explanations across ethically consequential datasets. ExplainBench provides unified wrappers for popular explanation algorithms, integrates end-to-end pipelines for model training and explanation generation, and supports evaluation via fidelity, sparsity, and robustness metrics. The framework includes a Streamlit-based graphical interface for interactive exploration and is packaged as a Python module for seamless integration into research workflows. We demonstrate ExplainBench on datasets commonly used in fairness research—COMPAS ( Angwin et al. 2016 ) , UCI Adult Income ( Kohavi 1996 ) , and LendingClub ( LendingClub 2020 ) and showcase how different explanation methods behave under a shared experimental protocol. By enabling reproducible, comparative analysis of local explanations, ExplainBench advances the methodological foundations of interpretable machine learning and facilitates accountability in real-world AI systems. 1 1 1 ExplainBench is available at https://github.com/jamesafful/explainbench Keywords: interpretability, explainable AI, SHAP, LIME, DiCE, benchmarking, fairness, local explanations 1 Introduction The deployment of machine learning (ML) models in high-stakes decision-making contexts has made interpretability and transparency a central concern in artificial intelligence (AI) research. From criminal justice and credit scoring to medical diagnostics and hiring processes, algorithmic systems are increasingly responsible for outcomes that have profound effects on individuals and society. As a result, the demand for interpretable models—systems whose decisions can be understood and scrutinized by humans—has grown substantially. However, despite the proliferation of interpretability methods such as LIME, SHAP, and counterfactual explanations, there remains a lack of standardization in how these methods are evaluated and compared, particularly in fairness-critical domains. Interpretability enables users to understand, trust, and contest algorithmic decisions. In domains such as recidivism prediction or income classification, the implications of incorrect or biased decisions are not merely theoretical—they are deeply ethical and political. For example, proprietary algorithms like COMPAS ( Angwin et al. 2016 ) have been shown to exhibit racial bias, with recidivism scores unfairly skewed against Black defendants. In such settings, black-box models without explanation mechanisms undermine both accountability and fairness. Interpretability serves as a bridge between model performance and social acceptability, allowing stakeholders—judges, doctors, loan officers, and affected individuals—to make informed decisions based on how and why a model arrived at its conclusion. Despite its growing importance, interpretability remains a nebulous concept. Unlike metrics such as accuracy or AUC-ROC, which are grounded in statistical theory, interpretability is subjective and context-dependent. Different explanation methods may yield divergent results even when applied to the same model and input instance. This variability makes it difficult to systematically assess and compare explanation methods across datasets, models, and fairness concerns. Existing interpretability libraries typically focus on individual explanation techniques, with little support for benchmarking, evaluation, or dataset diversity. Consequently, researchers and practitioners are left to cobble together ad hoc solutions, impeding progress toward standardized practices in interpretability evaluation. In light of these challenges, we argue that the field of interpretable machine learning lacks a reproducible and extensible framework for evaluating local explanations in fairness-sensitive applications. Local explanation methods—such as SHAP, LIME, and counterfactual explanations—aim to provide human-interpretable justifications for individual model predictions. However, the choice of method can influence both the content and quality of explanations, which in turn affects human decision-making, auditing processes, and downstream fairness interventions. A structured benchmark would facilitate more rigorous experimentation, foster transparency in evaluation, and guide the design of future interpretability methods. To address this gap, we introduce ExplainBench , an open-source framework for benchmarking local model explanations across real-world datasets involving socially sensitive decision-making. ExplainBench offers a unified interface for generating, visualizing, and evaluating explanations from state-of-the-art methods such as SHAP, LIME, and DiCE-based counterfactuals. The framework includes preprocessed datasets (COMPAS, UCI Adult Income, LendingClub), model training pipelines, explanation wrappers, and reproducible Jupyter notebooks. In addition, ExplainBench provides an interactive Streamlit dashboard for exploring explanations in an accessible and user-friendly manner. Unlike existing toolkits, ExplainBench is designed to support extensibility, interpretability evaluation, and cross-method comparison. Researchers can plug in new explanation methods, experiment with different model architectures, and evaluate explanations using metrics such as fidelity (how well the explanation approximates the black-box), sparsity (number of features involved), and stability (sensitivity to input perturbations). The framework’s modular design allows it to serve as both a pedagogical tool and a research testbed. ExplainBench includes datasets that have been widely discussed in the fairness literature. The COMPAS dataset, derived from U.S. court records, includes sensitive features such as race, age, and sex, and has been the subject of high-profile controversies regarding algorithmic bias. The UCI Adult Income dataset is a benchmark dataset for income classification and has been used extensively in studies of algorithmic discrimination. LendingClub, a dataset on peer-to-peer loan approvals, reflects real-world biases in credit allocation. By including these datasets, ExplainBench enables researchers to investigate how interpretability methods perform in contexts where fairness considerations are paramount. At the core of ExplainBench is a set of explanation wrappers that abstract away the idiosyncrasies of each interpretability method. Users can apply LIME, SHAP, or DiCE explanations with a common interface, allowing for streamlined experimentation. These wrappers are fully compatible with scikit-learn models and support local explanations at the individual instance level. The Streamlit interface provides interactive controls for dataset selection, instance navigation, explanation visualization, and feature contribution comparison. This front-end layer democratizes interpretability by making explanations accessible to users without extensive programming expertise. ExplainBench emphasizes reproducibility at every stage of the pipeline. All experiments are encapsulated in Jupyter notebooks, and the entire framework is packaged as a PyPI module, making it easy to install and extend. By adhering to open-source best practices and providing clear documentation, ExplainBench fosters transparency and facilitates community-driven development. Researchers can reproduce experiments, audit explanation methods, and contribute new datasets or evaluation metrics with minimal overhead. The release of ExplainBench comes at a pivotal moment in the evolution of responsible AI. Regulatory frameworks such as the EU AI Act and proposed U.S. algorithmic accountability policies are increasingly emphasizing the importance of explainability in automated decision-making systems. At the same time, recent critiques have highlighted the limitations of current interpretability methods, especially in high-stakes settings. ExplainBench offers a timely contribution by providing a robust platform for interpretability evaluation, helping bridge the gap between technical innovation and real-world accountability. The remainder of this paper is structured as follows. In Section 2, we survey existing work on interpretability methods and frameworks. Section 3 presents the architecture and design of ExplainBench, including its modular components and datasets. Section 4 describes case studies on COMPAS, UCI Adult, and LendingClub datasets, showcasing the use of SHAP, LIME, and DiCE explanations. In Section 5, we conduct a quantitative evaluation of explanation methods using fidelity, sparsity, and stability metrics. Section 6 discusses broader implications, limitations, and potential extensions. Finally, Section 7 concludes with a summary of contributions and directions for future work. 2 Related Work The pursuit