Revealing Multimodal Causality with Large Language Models
Jin Li, Shoujin Wang, Qi Zhang, Feng Liu, Tongliang Liu, Longbing Cao Shui Yu Fang Chen · 2025
Read Revealing Multimodal Causality with Large Language Models by Jin Li, Shoujin Wang, Qi Zhang, Feng Liu, Tongliang Liu, Longbing Cao Shui Yu Fang Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17784.txt.
Introduction
Jin Li and 7 co-authors use large language models to find cause links in mixed data. Their 2025 frame, MLLM-CD, works on text, image, and other modes at once. Pure data checks can miss how modes touch each other, and cause links can stay vague.
Three parts do the work in turns. A contrast step finds true factors from paired samples. A stats step maps cause links between those factors. A what-if step then uses model world facts to refine the map, round by round. Tests on fake and real data sets show the frame finds real factors and links.
Catalog record
| CREATOR | Jin Li, Shoujin Wang, Qi Zhang, Feng Liu, Tongliang Liu, Longbing Cao Shui Yu Fang Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 109 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17784
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revealing Multimodal Causality with Large Language Models Jin Li, Shoujin Wang, Qi Zhang, Feng Liu, Tongliang Liu, Longbing Cao Shui Yu Fang Chen Abstract Uncovering cause-and-effect mechanisms from data is fundamental to scientific progress. While large language models (LLMs) show promise for enhancing causal discovery (CD) from unstructured data, their application to the increasingly prevalent multimodal setting remains a critical challenge. Even with the advent of multimodal LLMs (MLLMs), their efficacy in multimodal CD is hindered by two primary limitations: (1) difficulty in exploring intra- and inter-modal interactions for comprehensive causal variable identification; and (2) insufficiency to handle structural ambiguities with purely observational data. To address these challenges, we propose MLLM-CD, a novel framework for multimodal causal discovery from unstructured data. It consists of three key components: (1) a novel contrastive factor discovery module to identify genuine multimodal factors based on the interactions explored from contrastive sample pairs; (2) a statistical causal structure discovery module to infer causal relationships among discovered factors; and (3) an iterative multimodal counterfactual reasoning module to refine the discovery outcomes iteratively by incorporating the world knowledge and reasoning capabilities of MLLMs. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of MLLM-CD in revealing genuine factors and causal relationships among them from multimodal unstructured data. The implementation code and data are available at https://github.com/JinLi-i/MLLM-CD . 1 Introduction Causal discovery (CD), which aims to infer the underlying causal structures from the data, is a fundamental task across numerous real-world scenarios, including healthcare [ 1 , 2 ] , finance [ 3 ] , and machine perception [ 4 , 5 , 6 ] , playing a vital role in advancing scientific inquiry and human cognition. As data generation and collection technologies [ 7 , 8 , 9 ] continue to evolve, real-world scenarios are increasingly characterized by multimodal and unstructured data, such as the combination of texts, images, and/or audio, making CD particularly crucial yet challenging in practice. Effectively tackling CD in such cases typically involves two key steps: (1) identifying potential causal variables (factor discovery) and their values from the raw unstructured data, and (2) inferring the causal relationships among these variables (structure discovery). Traditional CD methods [ 10 , 11 , 12 ] , though successful on structure discovery [ 13 ] , heavily rely on predefined causal variables in structured data and lack the inherent capability to perform factor discovery. As such, they are ill-equipped to handle unstructured data that lacks predefined variables, let alone the more complex multimodal unstructured data. Figure 1: Illustration of MLLM-CD in lung cancer diagnosis. It first employs contrastive factor discovery with MLLMs to identify potential causal variables and form structured data. Then, a CD algorithm is performed to infer causal structures. To further reduce ambiguities, it leverages MLLM’s world knowledge to generate multimodal counterfactual samples for iterative refinement. Identifying genuine causal variables from unstructured data is challenging and costly, as it often requires extensive manual feature engineering and domain expertise [ 14 ] . This makes it difficult to scale with the increasing volume and complexity of modern data. Recently, the advent of large language models (LLMs) [ 15 , 16 , 17 , 18 , 19 ] offers a promising opportunity to bridge this gap. With their remarkable capabilities in understanding context and processing natural language, LLMs can be leveraged to automate factor discovery from raw textual data [ 20 , 21 ] , paving the way for the following structure discovery. Integrating LLMs into CD pipelines holds significant potential to broaden the scope and applicability of conventional CD, yet this direction remains largely under-explored. To the best of our knowledge, COAT [ 20 ] is the only existing work that explicitly targets CD from unstructured data using LLMs, marking a significant step forward in this direction. It first extracts causal variables via LLMs and then infers the causal relationships with traditional CD methods. However, it is limited to unimodal data, typically text. In contrast, the rapid proliferation of multimedia content necessitates CD methods capable of effectively handling multimodal information [ 22 , 23 ] across various domains, such as medical diagnosis [ 24 ] , where clinical notes, medical images, and lab results collectively inform causal understanding [ 25 , 26 , 27 ] . While it may seem feasible to adapt unimodal approaches like COAT to multimodal scenarios using multimodal LLMs (MLLMs) [ 28 ] , our findings suggest that such a naïve adaptation falls short. As shown in Figure 2 , the adapted COAT struggles with two critical limitations on multimodal unstructured data: (1) it uncovers only a small set of causal factors, and (2) the inferred causal edges remain undirected. Figure 2: Results on MAG dataset 1 . (a) Ground truth. (b) Causal graph discovered by COAT. Visual and verbal causal variables are presented in blue and black. † † footnotetext: Results are based on Gemini 2.0. Please see Section 4.1 and Appendix D.3 for detailed experimental settings. Such limitations stem from two key challenges (CHs) inherent to multimodal CD which can not be addressed by simple adaptations. CH1. Probing Intra- and Inter-modal Interactions . Causal variables are often embedded in different modalities, and become identifiable only when interactions between them are probed (e.g., the observed interaction “smaller apples (image) have lower target scores (text)” helps reveal the variable of size ). Without guidance from such cross-modal interactions, existing methods tend to identify only the most salient factors (e.g., taste , aroma , and defects ), while overlooking more implicit yet equally important ones (e.g., nutrition ). CH2. Handling Structural Ambiguities . Inferring accurate causal structures from observational data alone is inherently challenging [ 29 ] , as multiple causal structures can yield the same statistical dependencies [ 13 ] , leading to undesired ambiguity. This issue is further amplified in multimodal settings since they typically involve more causal variables while the number of possible causal structures grows exponentially with the number of variables. To overcome these pressing challenges, we introduce MLLM-CD, a novel framework that integrates the strengths of MLLMs with conventional CD methods. As illustrated in Figure 1 , MLLM-CD is devised with three key modules. First, a contrastive factor discovery module is introduced for multimodal factor identification. It encourages the MLLM to explore intra- and inter-modal interactions through variations in semantically contrastive sample pairs, helping reveal implicit variables hidden in multimodal inputs. Second, causal structure discovery is employed to infer the causal structure among the identified factors. Third, a novel iterative multimodal counterfactual reasoning module is proposed to refine the inferred causal structures by reducing structural ambiguities. Specifically, by performing counterfactual reasoning on factors with uncertain relationships, it generates hypothetical yet causally consistent multimodal samples for iterative refinement. In this way, MLLMs offer counterfactual evidence beyond the observational data to mitigate structural ambiguities through their world knowledge and reasoning capabilities. Our main contributions are summarized as follows: • We propose MLLM-CD, the first work for causal discovery in multimodal unstructured data with a novel framework that integrates MLLMs into CD pipelines. This significantly extends the scope of CD beyond structured and/or unimodal data settings. • We introduce an innovative contrastive factor discovery module that encourages MLLMs to