Rethinking Explainability in the Era of Multimodal AI
Chirag Agarwal · 2025
Read Rethinking Explainability in the Era of Multimodal AI by Chirag Agarwal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13060.txt.
Introduction
Chirag Agarwal argues for a rethink of AI explanations in 2025. Modern AI systems blend text, images, graphs, and time data in one model. Their choices drive high stakes calls, so clear explanations matter for trust. Most explanation tools still treat each data type in isolation, though.
Single mode explanations miss how the modes sway each other inside the model. They can thus mislead users about why a call was made. Agarwal lays out principles for explanations that span modes. One principle tests influence by removing a mode and watching the shift. The paper urges the field to retire single mode readings for blended models.
Catalog record
| CREATOR | Chirag Agarwal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13060
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Explainability in the Era of Multimodal AI Chirag Agarwal Abstract While multimodal AI systems (models jointly trained on heterogeneous data types such as text, time series, graphs, and images) have become ubiquitous and achieved remarkable performance across high-stakes applications, transparent and accurate explanation algorithms are crucial for their safe deployment and ensure user trust. However, most existing explainability techniques remain unimodal, generating modality-specific feature-attribution, concepts, or circuit traces in isolation and thus failing to capture cross-modal interactions. This paper argues that such unimodal explanations systematically misrepresent and fail to capture the cross-modal influence that drives multimodal model decisions, and the community should stop relying on them for interpreting multimodal models . To support our position, we outline key principles for multimodal explanations grounded in modality: Granger-style modality influence (controlled ablations to quantify how removing one modality changes the explanation for another), Synergistic faithfulness (explanations capture the model’s predictive power when modalities are combined), and Unified stability (explanations remain consistent under small, cross-modal perturbations). This targeted shift to multimodal explanations will help the community uncover hidden shortcuts, mitigate modality bias, improve model reliability, and enhance safety in high-stakes settings where incomplete explanations can have serious consequences. 1 Introduction Recent advances in AI systems increasingly integrate multimodal data – vision and language, audio and language, graph and language, or videos, audios, and language – to achieve state-of-the-art performance ( Team et al. 2023 ; Hurst et al. 2024 ; Llama 2025 ) in complex multimodal tasks like visual-question answering ( Zhang et al. 2025 ) , multilingual automatic speech recognition ( Conneau et al. 2022 ) , video question-answering ( Patraucean et al. 2023 ) . For instance, multimodal clinical models combining imaging, lab results, and text outperform unimodal models by 6–30% margins ( Pahud de Mortanges et al. 2024 ) . With the increasing development of multimodal models, it becomes imperative that their predictions are readily explainable to relevant stakeholders. However, the field of explainable AI (XAI) has not kept pace with the multimodal surge and most existing explainability techniques remain unimodal , designed for models with a single input type – explaining an image classifier with a heatmap ( Ribeiro et al. 2016 ; Fong and Vedaldi 2017 ; Sundararajan et al. 2017 ; Selvaraju et al. 2020 ; Agarwal and Nguyen 2020 ) , identifying circuit traces in large language models (LLMs) ( Ameisen et al. 2025 ) , visualizing attention weights ( Galassi et al. 2020 ; Danilevsky et al. 2020 ) , or node- and edge-weights for graph models ( Ying et al. 2019 ; Agarwal et al. 2022c ; Agarwal et al. 2023 ) . To faithfully explain a multimodal model, we must develop modality-aware techniques that reveal how multiple modalities interact to produce their predictions and satisfy certain multimodal properties. We argue that the AI community must stop relying on unimodal explainability techniques in a multimodal world as they yield misleading/incomplete explanations, missing the core aspect of multimodal model behavior. For instance, in multimodal sentiment analysis, a model may draw on both spoken audio and transcript text; an unimodal explanation could indicate that positive words in the text led to a positive sentiment prediction, missing that the sarcastic tone in the audio drove the model’s negative classification. Present work. We outline why unimodal XAI fails for multimodal systems by examining model properties that unimodal explanations cannot capture. We illustrate these failures with case studies across different multimodal models (see Figs. 1 - 2 ), and provide key desiderata for multimodal explanations, viz. i) Granger-style modality influence, i.e., measuring how ablating one modality’s explanation reshapes another’s explanation, ii) Synergistic faithfulness, i.e., matching each modality’s true contribution to the model’s output, and iii) Unified stability, i.e., robustness under modality-specifc imperceptible perturbations, highlighting the urgent need for multimodal-aware explainability frameworks – a necessary step to keep our understanding of AI systems on pace with their growing complexity. Overall, in this work, we argue that relying on unimodal explainability for multimodal models is fundamentally flawed and demands a paradigm shift, revisiting the design of explanation algorithms, redefining evaluation protocols, and embracing truly multimodal evaluation criteria. 2 Why Unimodal Explanations Fall Short for Multimodal Models? Unimodal explanation techniques assume a single input source drives the model’s decisions and thus generate individual explanations for each modality. Common approaches include saliency maps highlighting important pixels in an image ( Ribeiro et al. 2016 ; Fong and Vedaldi 2017 ; Sundararajan et al. 2017 ; Selvaraju et al. 2020 ; Agarwal and Nguyen 2020 ) , developing concept bottleneck layers for implicit interpretability ( Koh et al. 2020 ) , attention-weight visualization ( Galassi et al. 2020 ; Danilevsky et al. 2020 ) , node and edge weights for graph machine learning models ( Ying et al. 2019 ; Agarwal et al. 2022c ; Agarwal et al. 2023 ) , and several other model-agnostic feature attribution methods that assign importance values to input features. These methods fall short in explaining multimodal models as they fail to capture several key properties of multimodal model behavior. We discuss these limitations below and why ignoring them leaves interpretations of multimodal systems fundamentally inaccurate. 2.1 Unimodal Explanations Fail to Capture the Multimodal Attribution Gap Most of the aforementioned explanation methods have been useful for explaining unimodal models in isolation ( Agarwal et al. 2022b ) . For instance, Grad-CAM heatmaps ( Selvaraju et al. 2020 ) can pinpoint which part of an image influenced an object recognition result, and attention-based explanations can indicate which words a language model focused on for a prediction, examining the relationship between a model’s prediction and the features of one modality at a time. However, for a multimodal model with inputs, say, from image I and text T , a unimodal approach might produce an importance map over pixels or a highlight of words – essentially treating the image and text separately. Such an approach treats explainability for multimodal models as “ explainability of parts ” rather than explaining the whole (see Figure 1 ), missing the fundamental fact that multimodal models are not merely the sum of two independent parts and a multi-modal model’s behavior cannot be fully explained by dissecting each modality’s influence in a silo. Unimodal explanations, by design, have no mechanism to represent such dependencies, and this limitation has practical consequences. This is further corroborated in Neo et al. 2024 , where the authors identify the most important tokens for the model’s decision using Integrated Gradients ( Sundararajan et al. 2017 ) and perform a token ablation experiment to impair the model’s ability to perform object identification. Their results show that, for a significant number of ablated tokens, object token ablation consistently results in larger performance decreases across different visual-language models as compared to the gradient-based baseline, highlighting the ineffectiveness of gradient-based explanations in a multimodal setting. Figure 1: Unimodal XAI (U-XAI) vs. Multimodal XAI (M-XAI) Attributions in Clinical Decision Support. (A) A multimodal diagnostic model that processes both a patient’s clinical text report (symptoms, physical exam) and their chest X-ray to conclude, “ The patient has Pneumonia, ” but a clinician, examining the case, notes no radiographic evidence of pneumonia. (B) Applying unimodal explainability method separately