Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
Shahin Atakishiyev, Housam K.B. Babiker, Jiayi Dai, Nawshad Farruque, Teruaki Hayashi, Nafisa Sadaf Hriti, Md Abed Rahman, Iain Smith, Mi-Young Kim, Osmar R. Zaïane, Randy Goebel · 2025
Read Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations by Shahin Atakishiyev, Housam K.B. Babiker, Jiayi Dai, Nawshad Farruque, Teruaki Hayashi, Nafisa Sadaf Hriti, Md Abed Rahman, Iain Smith, Mi-Young Kim, Osmar R. Zaïane, Randy Goebel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17256.txt.
Introduction
Shahin Atakishiyev and ten co-authors review how to explain large language models in this 2025 paper. Users cannot see why a model picks its next word. Models also make false claims that look sure and smooth. Both flaws make trust hard in high-stakes fields.
The review sorts local tricks that explain one output at a time. It also sorts work that maps inner parts to clear roles. Case work in health care and self-driving shows how trust shifts. Lay readers weigh such notes quite unlike trained experts. The authors close with open gaps and paths to more honest notes.
Catalog record
| CREATOR | Shahin Atakishiyev, Housam K.B. Babiker, Jiayi Dai, Nawshad Farruque, Teruaki Hayashi, Nafisa Sadaf Hriti, Md Abed Rahman, Iain Smith, Mi-Young Kim, Osmar R. Zaïane, Randy Goebel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17256
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations Shahin Atakishiyev, Housam K.B. Babiker, Jiayi Dai, Nawshad Farruque, Teruaki Hayashi, Nafisa Sadaf Hriti, Md Abed Rahman, Iain Smith, Mi-Young Kim, Osmar R. Zaïane, Randy Goebel Abstract Large language models have exhibited impressive performance across a broad range of downstream tasks in natural language processing. However, how a language model predicts the next token and generates content is not generally understandable by humans. Furthermore, these models often make errors in prediction and reasoning, known as hallucinations. These errors underscore the urgent need to better understand and interpret the intricate inner workings of language models and how they generate predictive outputs. Motivated by this gap, this paper investigates local explainability and mechanistic interpretability within Transformer-based large language models to foster trust in such models. In this regard, our paper aims to make three key contributions. First, we present a review of local explainability and mechanistic interpretability approaches and insights from relevant studies in the literature. Furthermore, we describe experimental studies on explainability and reasoning with large language models in two critical domains — healthcare and autonomous driving — and analyze the trust implications of such explanations for explanation receivers. Finally, we summarize current unaddressed issues in the evolving landscape of LLM explainability and outline the opportunities, critical challenges, and future directions toward generating human-aligned, trustworthy LLM explanations. \forestapplylibrarydefaults linguistics 1 Introduction Pretrained large language models (LLMs) have emerged as powerful tools for a range of complex natural language processing (NLP) applications in the field of artificial intelligence (AI). Primarily based on the Transformer architecture vaswani2017attention , recent LLMs, such as the family series from GPT brown2020language ; achiam2023gpt ; gpt5systemcard2025 , Vicuna vicuna2023 , LLaMA touvron2023llama ; touvron2023llama2 , Mistral jiang2023mistral , Claude anthropic_claude ; anthropic2025claude45 , Qwen bai2023qwen ; team2024qwen2 ; yang2025qwen3 , Gemini team2023gemini ; comanici2025gemini , DeepSeek bi2024deepseek ; liu2024deepseek ; liu2024v3deepseek ; guo2025deepseek , Grok xai2024grok ; xai2024grok2 ; xai2025grok4 and Kimi team2025kimi1.5 ; team2025kimik2 have demonstrated impressive performance across many NLP tasks, and the list continues to expand. These models typically use a Transformer-based deep neural network as a base model and are augmented with an extended number of parameters, a larger volume of data, and an increased overall model size. Numerous empirical studies show that larger-size LLMs substantially outperform smaller models in terms of their emergent abilities, such as reasoning, instruction following, program execution, and general-purpose task-solving capabilities brown2020language ; driess2023palm ; wei2022emergent . While LLMs’ emergent abilities are impressive at first glance, there are several issues with these tools singh2024rethinking . First, almost all state-of-the-art LLMs have a common problem of producing hallucinations , i.e., they produce convincing but falsified, factually incorrect content. Depending on the context, LLM-generated false statements can be detected easily; however, there are also cases where such statements may seem correct while actually not being evidence-grounded or confirmed without careful judgment huang2025survey ; sun2025and . Furthermore, LLMs might have biases in their generated content while responding to a specific prompt, such as in politics, culture, race, and gender-based topics haller2023opiniongpt . Finally, LLMs remain “black-box" models in general; people do not understand how the internal knowledge base and inner working mechanism enable these models to make their predictions. These problems hinder the applicability of such black-box AI models in safety-critical tasks and domains, and raise serious issues regarding their societal alignment and compliance with regulatory principles gabriel2020artificial . Consequently, there is an imminent need for a thorough investigation of interpretability and explainability in LLMs. The topics of interpretability and explainability have been used interchangeably or as distinguished from each other in explainable artificial intelligence (XAI), depending on the use case and the nature of the tasks where they are considered kim2021multi ; saeed2023explainable . In the context of LLMs, those topics can be explored in two directions: Understanding these models 1) by their inner workings, i.e., mechanistic interpretability, and 2) based on the predictions they generate for particular tasks, i.e., local explainability (Figure 1 ). Our paper tries to elucidate both of these directions and specifically aims to underscore the implications of local and mechanistic explanations from the trust perspective for the targeted interaction partners. Given that interpretability and explainability in LLMs remain in their early stages of development, we begin by elucidating the foundational concepts within these tools and synthesizing the key insights and findings emerging from recent research within the LLM community. After analyzing the findings from the reviewed studies, we further summarize insights from empirical evidence in two safety-critical applications. Finally, building on our critical analysis and experimental results, we propose future directions for advancing the trustworthiness of LLM explanations. 1.1 Contributions Focusing on LLM interpretability and explainability through the lenses of trustworthiness and human-centricity, our paper makes three key contributions: • We review the local explainability and mechanistic interpretability techniques in the LLM literature, present insights from these studies, and reveal critical aspects of trustworthy LLM explanations; • We draw insights from experiments in two safety-critical domains and show trust implications of LLM explanations for explanation receivers; • We present unaddressed issues in the current approaches to LLM explainability and propose guidelines toward achieving eight fundamental goals of a trustworthy LLM with explanations. 1.2 Structure of the Paper The paper is organized into the following sections. After the introduction, Section 2 presents background information on LLMs. Section 3 reviews related studies on LLM explainability and shows how our paper bridges the gap in the current literature. Section 4 presents a detailed review of local explainability and mechanistic interpretability approaches in LLMs. Critical aspects in presenting trustworthy LLM explanations and their assessment are covered in Sections 5 and 6, respectively. After that, Section 7 presents empirical insights from two safety-critical domains, and key findings from these studies in conjunction with the existing literature are presented in Section 8. Finally, Section 9 proposes potential future directions for advancing trustworthiness in LLM explanations, and Section 10 concludes the paper with an overall summary. Figure 1 : Local explainability and mechanistic interpretability approaches in LLMs. 2 Background of Language Models 2.1 The Rise of LLMs The history of LLMs dates back to the early 1980s and all the way through 2025, with several evolutions based on their training mechanisms and capabilities. Unsurprisingly, all these models are based on the main theme of language modeling, i.e., predicting the probability of a word occurring given its surrounding context. Earlier statistical learning models brown1992class ; ney1994structuring were developed on a purely Markov assumption, i.e., the probability of a word occurrence is wholly dependent on the n-grams preceding it. However, these methods were not successful in predicting words accurately when the preceding context is large, which creates a huge computational burden of calculating an exponential number of transition probabilities. Subsequently, deep learning models that can learn distributed word representations more efficiently have been shown to exploit the back-propagation technique proposed by rumelhart1986learning ; bengio2000neural . Later, these methods were improved by posing them as a self-supervised classification task with a contrastive loss function and negative sampling; this improvement led to the development of language models such as Word2Vec mikolov2013distributed and FastText bojanowski2017enriching . The resulting distributed representations primarily focus on how a