A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems

Alessandro Gambetti, Qiwei Han, Hong Shen, Cláudia Soares · 2025

Read A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems by Alessandro Gambetti, Qiwei Han, Hong Shen, Cláudia Soares (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09849.txt.

Introduction

Gambetti and co-authors survey how explainable AI for medical decision support is judged by the humans meant to trust it. They gathered thirty-one human-centered evaluations and sorted them by explanation method, study design, and adoption barrier. The field leans heavily on after-the-fact, model-free explainers such as SHAP and Grad-CAM, over eighty percent of studies.

Clinician samples were tiny, under twenty-five in most studies. Explanations usually raised trust and confidence, yet often raised mental workload too and clashed with how doctors actually reason. The authors propose a stakeholder-first framework, blending social and technical lenses, to steer explainers toward tools clinics can truly use.

Catalog record

CREATORAlessandro Gambetti, Qiwei Han, Hong Shen, Cláudia Soares
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09849

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Survey on Human-Centered Evaluation of Explainable AI Methods in Clinical Decision Support Systems Alessandro Gambetti, Qiwei Han, Hong Shen, Cláudia Soares Abstract Explainable AI (XAI) has become a crucial component of Clinical Decision Support Systems (CDSS) to enhance transparency, trust, and clinical adoption. However, while many XAI methods have been proposed, their effectiveness in real-world medical settings remains underexplored. This paper provides a survey of human-centered evaluations of Explainable AI methods in Clinical Decision Support Systems. By categorizing existing works based on XAI methodologies, evaluation frameworks, and clinical adoption challenges, we offer a structured understanding of the landscape. Our findings reveal key challenges in the integration of XAI into healthcare workflows and propose a structured framework to align the evaluation methods of XAI with the clinical needs of stakeholders. 1 Introduction Artificial Intelligence (AI) is increasingly being integrated into Clinical Decision Support Systems (CDSS) to improve timely diagnosis, optimize treatment plans, and reduce healthcare costs Sendak et al. 2020 . Estimates suggest that AI adoption in healthcare could reduce U.S. healthcare expenditures by 5-10%, translating to $200–$360 billion in annual savings while maintaining quality and access Sahni et al. 2023 . The rapid growth of AI-driven CDSS has also attracted significant investment, with venture capital funding in AI in health exceeding $11 billion in 2024 Goldsack and Overgaard 2024 . These trends indicate that the role of AI in medical decision making will continue to expand. However, a key challenge accompanying this expansion is the necessity for trust and transparency in AI-driven medical decisions. Despite advancements in deep learning and other black-box AI models, clinicians often hesitate to adopt AI-generated recommendations due to their lack of interpretability Hou et al. 2024 . Trust in CDSS is critical because errors in automated decision-making can have life-threatening consequences Zhang and Zhang 2023 . In response, Explainable AI (XAI) has emerged as a solution to improve model transparency, inderectly allowing medical professionals to better understand why an AI system makes certain predictions Loh et al. 2022 . Additionally, regulatory frameworks, such as the General Data Protection Regulation (GDPR), emphasize the need for explainability in automated decision-making, thereby necessitating interpretable AI solutions in healthcare. Despite extensive research in XAI for healthcare Loh et al. 2022 , a critical gap remains: there is limited systematic analysis of how XAI explanations impact real-world clinical adoption and decision-making Ghassemi et al. 2021 . While prior work has primarily focused on developing new XAI techniques, fewer studies have systematically examined whether these methods effectively support clinicians in practice Amann et al. 2022 . Furthermore, the evaluation of XAI in healthcare is highly fragmented, relying on inconsistent methodologies that range from proxy simulations to real-world clinical trials Doshi-Velez and Kim 2017 . This raises a fundamental question: How should XAI techniques be evaluated to ensure their usability, trustworthiness, and effectiveness in real clinical settings? To address this gap, this paper presents a systematic survey of human-centered evaluations of XAI methods in CDSS, focusing on the intersection of explainability, usability, and clinical decision-making. Specifically, we: (1) categorize existing works based on XAI methodologies, evaluation frameworks, and clinical adoption challenges, offering a structured synthesis of the field; (2) identify key gaps in current evaluation practices from clinicians perceptions of XAI-based CDSS, revealing that while XAI is generally well-received, significant challenges remain—such as high cognitive load, misalignment with clinical knowledge, and lack of early involvement with stakeholders; (3) introduce a conceptual framework to bridge the socio-technical gap in AI-driven CDSS by aligning XAI evaluation methods with the needs of diverse stakeholders ( e.g. , clinicians, hospital administrators, AI developers); (4) propose a medical field-agnostic, stakeholder-centric approach for CDSS development, advocating that CDSS should be designed as human-augmentation tools rather than replacements. To theoretically ground this misalignment between human expectations and technological capabilities, we leverage Ackerman’s socio-technical gap theory, which highlights the challenges arising from such discrepancies Ackerman 2000 . This analysis informs our proposed framework, emphasizing stakeholder-driven evaluations to ensure CDSS solutions are clinically relevant, interpretable, and effectively integrated into healthcare workflows. By aligning technical advancements in XAI with rigorous human-centered evaluations, we aim to steer research toward developing trustworthy, deployable CDSS solutions. We organize the paper as follows. Section 2 provides background on CDSS, XAI methods, and human-centered evaluations. Section 3 presents a structured review of existing works in this field, categorizing them based on XAI methodologies, evaluation frameworks, and clinical adoption challenges. Section 4 discusses the socio-technical challenges and proposes a conceptual framework to bridge the gap between XAI methodologies and stakeholder needs, concluding pointing out future research directions. 2 Background 2.1 Clinical Decision Support Systems (CDSS) CDSS have long been integral to medical decision-making, assisting clinicians in enhancing patient outcomes. Initially emerging in the late 1950s with rule-based systems, CDSS evolved significantly with advancements in AI. Modern CDSS now leverage sophisticated machine learning algorithms to process vast datasets efficiently. Concurrently, the rise of XAI has enabled the integration of interpretable solutions, essential for fostering trust and promoting the adoption of CDSS in clinical practice Ghassemi et al. 2021 . Operationally, CDSS utilize patient data to generate specific treatment recommendations Musen et al. 2021 . The back-end employs machine learning algorithms trained for specific tasks, processing data from Electronic Health Records (EHRs). EHRs include both structured data (e.g., tabular data) and/or unstructured data (e.g., text, images), representing comprehensive digital versions of patients’ medical histories, including diagnoses, treatments, and test results. System performance is evaluated using machine learning metrics pertinent to the specific task. Synchronously, the front-end runs a user interface (UI) connected to the back-end that clinical practitioners are supposed to interact with. Such UI displays patient data, alerts, recommendations (sometimes with explanations), and decision-support reports in an accessible manner to facilitate clinical decision-making. To this end, human-centered evaluations are conducted to assess the overall effectiveness and efficiency of the system in the specific task. However, researchers have identified that the absence of such evaluations is a primary factor in the lack of adoption of AI-based CDSS solutions Musen et al. 2021 . In summary, the integration of XAI together with human-centered evaluation assures that the system is usable within clinical workflows and more transparently meets the needs of stakeholders to foster acceptance of CDSS. 2.2 Explainability and XAI Methods: a Taxonomy From a psychological standpoint, humans seek explanations to predict future events, as explanations facilitate generalization Vasil and Lombrozo 2022 . Unlike descriptions, explanations provide understanding by identifying “difference-makers” in causal relationships. In medicine, where transparency and accountability are crucial, explanations may offer insights into the rationale behind AI-based CDSS recommendations, enabling practitioners to align the model’s reasoning with their medical expertise. In AI, explainability and interpretability improve people’s understanding of such models, driving from suboptimal to “better” outcomes Boyd-Graber et al. 2022 . More in general, the main principle of XAI is to make the decision-making process of AI systems transparent and understandable to humans. Many taxonomies have been proposed to categorize XAI methodologies ( e.g. , see Speith 2022 ). In this paper, we adopt the conceptual approach based on the following macro criteria (for a more comprehensive review, please refer to Molnar 2022 ): • Type : Intrinsic (or Ante Hoc) or Post Hoc ;

Read the complete text