TT-XAI: Trustworthy Clinical Text Explanations via Keyword Distillation and LLM Reasoning

Kristian Miok, Blaz Škrlj, Daniela Zaharie, Marko Robnik Šikonja · 2025

Read TT-XAI: Trustworthy Clinical Text Explanations via Keyword Distillation and LLM Reasoning by Kristian Miok, Blaz Škrlj, Daniela Zaharie, Marko Robnik Šikonja (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08273.txt.

Introduction

Kristian Miok and co-authors make clinical AI explain itself by first shrinking the medical record to its keywords, in work from 2025. Language models reading long, messy discharge notes often predict opaquely, and their explanations drift.

The TT-XAI framework distills each note into domain keywords. A classifier trained on these digests performed better, and local explanations of its choices grew sharper. The keywords also steered a large language model to write shorter, more relevant chains of reasoning. Deletion tests, model self-scoring, and a blinded expert study all favored the keyword method. The authors offer the pipeline as a scalable route to auditable clinical decision support.

Catalog record

CREATORKristian Miok, Blaz Škrlj, Daniela Zaharie, Marko Robnik Šikonja
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08273

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TT-XAI: Trustworthy Clinical Text Explanations via Keyword Distillation and LLM Reasoning Kristian Miok, Blaz Škrlj, Daniela Zaharie, Marko Robnik Šikonja Abstract Clinical language models often struggle to provide trustworthy predictions and explanations when applied to lengthy, unstructured electronic health records (EHRs). This work introduces TT-XAI , a lightweight and effective framework that improves both classification performance and interpretability through domain-aware keyword distillation and reasoning with large language models (LLMs). First, we demonstrate that distilling raw discharge notes into concise keyword representations significantly enhances BERT classifier performance and improves local explanation fidelity via a focused variant of LIME. Second, we generate chain-of-thought clinical explanations using keyword-guided prompts to steer LLMs, producing more concise and clinically relevant reasoning. We evaluate explanation quality using deletion-based fidelity metrics, self-assessment via LLaMA-3 scoring, and a blinded human study with domain experts. All evaluation modalities consistently favor the keyword-augmented method, confirming that distillation enhances both machine and human interpretability. TT-XAI offers a scalable pathway toward trustworthy, auditable AI in clinical decision support. 1 Introduction Deep pre-trained language models (PLMs), such as BERT family of transformer models, and their adaptations to medical texts, have become essential tools in clinical natural language processing (NLP), demonstrating strong performance across tasks like diagnosis classification, medical coding, and clinical event prediction. These models can process unstructured free-text data at scale, offering opportunities to support clinical workflows and improve patient care. However, despite their predictive power, PLMs typically lack interpretability—an essential requirement in high-stakes domains such as healthcare, where decisions must be transparent, justifiable, and auditable by medical professionals. For many medical applications, their transparency is also a legal requirement, e.g., by the EU AI Act. To address this, post-hoc perturbation-based explanation techniques such as LIME ( Ribeiro et al. 2016 ) and SHAP ( Lundberg and Lee 2017 ) are used to highlight input features that influence model predictions. While effective in some domains, these methods encounter substantial limitations when applied to the lengthy, heterogeneous nature of electronic health records (EHRs). Clinical notes often exceed thousands of tokens and contain dense medical terminology, irregular formatting, and non-informative boilerplate. In this setting, perturbation-based explanation methods may generate ungrammatical, semantically incoherent samples that produce diffuse and clinically meaningless feature attributions, limiting their usefulness for medical reasoning and model auditing. The task of interpreting clinical NLP models becomes especially critical in conditions that exhibit variable severity and unpredictable outcomes. One such condition is nephrolithiasis (kidney stone disease), a prevalent urological issue affecting over 10% of adults worldwide Scales et al. 2012 . While many cases are managed in outpatient settings, others lead to prolonged hospitalization due to complications such as infection, obstruction, or renal failure ( Pearle et al. 2014 ) . Early identification of patients at risk for extended length of stay (LOS) can inform triage decisions, allocate resources more efficiently, and guide personalized treatment strategies. However, structured EHR fields often fail to capture the nuanced clinical indicators associated with complicated kidney stone cases. Instead, relevant signals are typically embedded within long narrative notes, which pose challenges for both prediction and interpretability. To support trustworthy and clinically useful predictions in this context, it is necessary to improve both model performance and the clarity of post-hoc explanations. Specifically, there is a need for approaches that can filter or distill complex EHR narratives into more focused representations retaining essential clinical content while enabling both accurate classification and interpretable reasoning. Furthermore, the ability to generate and evaluate multi-step explanations (e.g., via chain-of-thought reasoning) using large language models (LLMs) may provide an additional layer of interpretability, particularly when explanations are guided by clinically salient information. Contributions. This paper presents a comprehensive study of explainability and performance in clinical NLP models for predicting prolonged hospitalization in kidney stone patients. Our key contributions are: 1. Assessment of BERT limitations in clinical settings. We analyze the shortcomings of BERT-based models when applied to raw discharge summaries, highlighting challenges in both predictive robustness and explanation quality due to the length, noise, and redundancy of clinical notes. 2. Keyword-based distillation for performance and interpretability. We demonstrate that distilling notes into domain-relevant keyword representations significantly improves classification performance and enables more faithful and focused LIME explanations by restricting perturbations to clinically salient terms. 3. Enhancing reasoning via keyword-guided LLM prompts. We propose a chain-of-thought prompting strategy that combines keywords with full-text input to guide LLM-based explanation generation. This method produces more structured, concise, and clinically useful reasoning chains than full-text prompts alone. 4. Structured LLM and human expert evaluation. We introduce a dual evaluation framework using both LLM-based scoring and blinded human specialist assessment. Results confirm that keyword-augmented reasoning is consistently rated higher in clarity and clinical relevance, reinforcing the trustworthiness of our approach. Paper Structure. The remainder of this paper is organized as follows. In Section 2 , we review related work on interpretability in NLP, clinical explainability, text distillation, and LLM-based reasoning. Section 3 presents our methods, including the dataset, keyword extraction pipeline, classification model, and explanation strategies. In Section 4 , we present empirical results on classification performance, explanation fidelity, and LLM-based reasoning evaluation. Section 5 discusses the implications and limitations of our approach. We conclude in Section 6 with final remarks and directions for future work. 2 Related Work We outline related work divided into four subsections. We first present the topic of interpretability in NLP and then drill in into interpretability in clinical NLP. We continue with knowledge distillation in healthcare through text summarization and by using LLMs for explanation and reasoning. We end the section by outlining the main challenges of explainability in clinical setting and differences of our approach to existing works. 2.1 Interpretability in NLP Interpretability remains a central concern in modern NLP, especially as deep neural architectures grow in complexity. Among post-hoc explanation methods for tabular data, LIME ( Ribeiro et al. 2016 ) and SHAP ( Lundberg and Lee 2017 ) are widely used to generate local feature attributions by perturbing inputs and fitting interpretable surrogate models. These methods can also be used on text classifiers if the input is represented as features, e.g., in the TF-IDF weighted bag-of-words representation. While these methods are model-agnostic and effective on short, structured inputs, their reliability diminishes in long-form text due to perturbations that break grammatical structure or distort semantics ( Jacovi and Goldberg 2020 ) . In response, several alternatives have been proposed, including gradient-based saliency methods ( Li et al. 2016 ) , attention-based rationales ( Jain and Wallace 2019 ) , and counterfactual generation ( Goyal et al. 2019 ) , though none are free from limitations. The challenge is further magnified in clinical text, where interpretability must not only be accurate but also clinically meaningful to domain experts. 2.2 Interpretability in Clinical NLP In healthcare, the importance of trustworthy explanations is amplified by the risk and regulatory demands of clinical decision-making. Prior work has adapted LIME and SHAP for tasks such as ICU mortality prediction ( Caruana et al. 2015 ) , clinical note classification ( Bhatt et al. 2020 ) , and phenotyping from EHRs ( Choi et al. 2016 ) . However, long notes and ambiguous terminology often result in noisy or spurious attributions. Some studies have explored integrating domain-specific

Read the complete text