Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery
ChengAo Shen, Zhengzhang Chen *, Dongsheng Luo, Dongkuan Xu, Haifeng Chen, Jingchao Ni * · 2024
Read Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery by ChengAo Shen, Zhengzhang Chen *, Dongsheng Luo, Dongkuan Xu, Haifeng Chen, Jingchao Ni * (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13667.txt.
Introduction
ChengAo Shen and co-authors published a 2024 paper on finding cause-and-effect links with help from language models and mixed data. Classic methods for causal discovery rely on numbers alone and miss the meaning inside text and other records. Language models can read those meanings, but tools for causal work with them lag behind, especially when data comes in several forms.
Their system, MATMCD, uses two cooperating AI agents. One agent fetches and prepares extra data from varied sources. The other reasons over the combined material and proposes cause-and-effect constraints. Tools plug into both agents to ground their work. The authors tested the system on seven datasets. Results suggest that adding other data types sharpens causal discovery beyond what numbers alone give. The paper points toward decision tools that use everything a record holds, not just its measurements.
Catalog record
| CREATOR | ChengAo Shen, Zhengzhang Chen *, Dongsheng Luo, Dongkuan Xu, Haifeng Chen, Jingchao Ni * |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13667
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Multi-Modal Data with Tool-Augmented LLM Agents for Precise Causal Discovery ChengAo Shen, Zhengzhang Chen *, Dongsheng Luo, Dongkuan Xu, Haifeng Chen, Jingchao Ni * Abstract Causal discovery † † * Corresponding authors. is an imperative foundation for decision-making across domains, such as smart health, AI for drug discovery and AIOps. Traditional statistical causal discovery methods, while well-established, predominantly rely on observational data and often overlook the semantic cues inherent in cause-and-effect relationships. The advent of Large Language Models (LLMs) has ushered in an affordable way of leveraging the semantic cues for knowledge-driven causal discovery, but the development of LLMs for causal discovery lags behind other areas, particularly in the exploration of multi-modal data. To bridge the gap, we introduce MatMcd , a multi-agent system powered by tool-augmented LLMs. MatMcd has two key agents: a Data Augmentation agent that retrieves and processes modality-augmented data, and a Causal Constraint agent that integrates multi-modal data for knowledge-driven reasoning. The proposed design of the inner-workings ensures successful cooperation of the agents. Our empirical study across seven datasets suggests the significant potential of multi-modality enhanced causal discovery. 1 1 1 The code of MatMcd is available at https://github.com/D2I-Group/matmcd . 1 Introduction Identifying cause-and-effect relationships in complex systems is crucial for a variety of applications, including neuralgia diagnosis in medicine Tu et al. (2019) , protein pathway analysis in computational biology Sachs et al. (2005) , and root cause locating in microservice architectures Wang et al. (2023a) . These causal insights significantly benefit emerging fields such as smart health, AI-driven drug discovery, and AIOps. The process of discovering such relationships from observational data, known as causal discovery Yu et al. (2025) , typically generates a Directed Acyclic Graph (DAG). In this graph, edges represent the existence and direction of causal relationships between variables, as illustrated in Fig. 1 (a). This DAG not only governs the data generation process but also enhances the understanding of inter-variable influences, serving as the foundation for many downstream decision-making tasks Nguyen et al. (2023) . As such, constructing accurate causal graphs is essential to the reliability of subsequent analyses. Figure 1: An illustration of the general process of LLM based causal discovery: (a) a causal graph is estimated by an SCD algorithm; (b) a prompt is generated w.r.t. the graph; (c) LLMs reason about the causal structures; (d) the graph is refined based on the LLM response. Conventional methods primarily rely on data-driven statistical causal discovery (SCD), which can be categorized as non-parametric Spirtes and Glymour (1991) ; Silander and Myllymäki (2006) ; Huang et al. (2018) ; Xie et al. (2020) and semi-parametric Shimizu et al. (2006) ; Shimizu et al. (2011) ; Zheng et al. (2018) ; Tu et al. (2022) . These methods estimate causal relationships by analyzing the observational data of variables, but they overlook the semantic and contextual cues of the variables, resulting in suboptimal outcomes Takayama et al. (2024) . Commonsense and domain knowledge are invaluable for identifying cause-and-effect relationships among semantically meaningful variables. In light of this, growing research attention has been drawn to elicit such information for causal discovery. Large Language Models (LLMs), as praised by its astonishing reasoning ability drawing on extensive knowledge acquired from large-scale training Brown et al. (2020) ; Achiam et al. (2023) , now become a promising and cost-effective source of expert knowledge to aid causal discovery. For example, by merely prompting with variable names and some contextual cues, LLMs have been shown to infer meaningful causal relationships Ban et al. (2023) ; Jiralerspong et al. (2024) . More recently, hybrid approaches were introduced to combine LLMs with data-driven SCD algorithms, achieving enhanced accuracy in causal discovery Takayama et al. (2024) ; Khatibi et al. (2024) . However, most existing methods have yet to fully harness the potential of modern LLMs, particularly the agent systems built upon tool-augmented LLMs. An LLM agent is typically equipped with memory, reasoning, planning, and access to external tools such as calculators, search engines, and code compilers, rendering it superior in problem-solving compared to vanilla LLMs Yao et al. (2022) . As a single-agent system may be hallucination-prone even with self-reflection Li et al. (2023) ; Shinn et al. (2024) , multi-agent systems were introduced, which rivals more advanced models by combining multiple weaker agents. Despite the fast progress, LLM agents for causal discovery remains underexplored. The most relevant effort to date introduced a multi-agent system with debating LLMs for synergistic causal discovery Le et al. (2024) . However, this approach has never explored the potential of multi-modality in data – a feature that agent systems are well-equipped to handle. As Fig. 1 shows, hybrid methods for causal discovery typically prompt LLMs with the prior causal graph produced by some SCD algorithm, appended by some meta-data ( e.g. , variable names, dataset titles) as contexts. However, these inputs may fall short in fully activating the reasoning ability of LLMs. Inspired by the observation that abundant semantic data from external sources, such as webs and logs, can serve as an additional modality to the observational data (e.g., time series) for improving prompts, we propose a M ulti- A gent system with T ool-augmented LLMs for exploring M ulti-modal data enhanced C ausal D iscovery ( MatMcd ). Specifically, MatMcd is designed as a framework for refining causal graphs generated by SCD algorithms, involving two key agents: (1) a Data Augmentation Agent ( DA-agent , \lx@sectionsign 3.2 ), and (2) a Causal Constraint Agent ( CC-agent , \lx@sectionsign 3.3 ). Given a causal graph output by some SCD algorithm, DA-agent integrates meta-data ( i.e. , variable names and dataset titles) and calls tools such as web search APIs or offline log lookup APIs for iterative, reflection-based retrieval of augmented (textual) data, which is summarized as a compact cue in a different modality from the graphs. Upon receiving the augmented data, CC-agent combines it with the topological structure of the prior causal graph to infer the causal relationships among variables. Both agents comprise multiple cooperative LLMs with well-crafted mechanisms to handle sub-tasks such as tool-calling, memorizing, reasoning, and summarization. Retrieval Augmented Generation (RAG) Lewis et al. (2020) components are used where efficient memory is essential. In our experiments, we compared MatMcd with state-of-the-art (SOTA) baseline methods across five benchmark datasets and two public AIOps datasets of microservice systems. The results demonstrate substantial improvements in causal discovery by incorporating multi-modal data. The main contributions of this work are summarized as follows: • We propose to explore the problem of multi-modal data enhanced causal discovery via LLM agents, which is significant yet less studied. • We introduce MatMcd , a novel framework of multi-agent, as a testbed for assessing the effectiveness of multi-modal data in causal discovery. • We perform extensive experiments on a variety of datasets, where MatMcd reduces causal discovery errors (NHD) by up to 66.7% and improves root cause locating (MAP@10) by up to 83.3% over the best baselines, suggesting the potential of multi-modal data in causal discovery. 2 Related Work Causal Discovery Methods are mostly conventional data-driven SCD algorithms, including non-parametric methods Spirtes and Glymour (1991) ; Chickering (2002) ; Silander and Myllymäki (2006) ; Huang et al. (2018) ; Xie et al.