CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
Aneesh Komanduri, Karuna Bhaila, Xintao Wu · 2025
Read CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models by Aneesh Komanduri, Karuna Bhaila, Xintao Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11034.txt.
Introduction
Aneesh Komanduri and co-authors build, in a 2025 paper, a benchmark that tests whether vision-language AI can reason about cause and effect in pictures. Models that mix images and text now ace recognition and question answering, yet causal skills like spotting what interventions change have barely been probed. Their CausalVLBench sets three tasks: infer the causal structure behind a scene, predict which element an intervention should target, and forecast outcomes under a what-if change.
The team runs leading open-source models through the benchmark across three causal datasets. Strengths and weaknesses come out plainly: performance on the three tasks varies, and the gaps suggest current models lean on pattern matching more than on causal understanding. The authors offer the suite as a yardstick for a skill that safety-critical vision systems will need.
Catalog record
| CREATOR | Aneesh Komanduri, Karuna Bhaila, Xintao Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11034
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models Aneesh Komanduri, Karuna Bhaila, Xintao Wu Abstract Large language models (LLMs) have shown remarkable ability in various language tasks, especially with their emergent in-context learning capability. Extending LLMs to incorporate visual inputs, large vision-language models (LVLMs) have shown impressive performance in tasks such as recognition and visual question answering (VQA). Despite increasing interest in the utility of LLMs in causal reasoning tasks such as causal discovery and counterfactual reasoning, there has been relatively little work showcasing the abilities of LVLMs on visual causal reasoning tasks. We take this opportunity to formally introduce a comprehensive causal reasoning benchmark for multi-modal in-context learning from LVLMs. Our CausalVLBench encompasses three representative tasks: causal structure inference, intervention target prediction, and counterfactual prediction. We evaluate the ability of state-of-the-art open-source LVLMs on our causal reasoning tasks across three causal representation learning datasets and demonstrate their fundamental strengths and weaknesses. We hope that our benchmark elucidates the drawbacks of existing vision-language models and motivates new directions and paradigms in improving the visual causal reasoning abilities of LVLMs. We make our code and data available at https://github.com/Akomand/CausalVLBench . 1 Introduction With a growing emphasis on developing pre-trained models that emulate human reasoning and thinking patterns, a wide range of literature has also focused on assessing these models on complex reasoning tasks Nie et al. 2023 ; Mitchell et al. 2023 . As human intelligence is often hallmarked by causal reasoning, i.e., the ability to distinguish cause and effect, researchers have also prioritized evaluation of pre-trained models on causal inference tasks Zhang et al. 2023a ; Kıcıman et al. 2023 ; Jin et al. 2023 . The evaluation tasks vary among causal effect inference Jin et al. 2023 , causal discovery Jiralerspong et al. 2024 ; Vashishtha et al. 2023 , and counterfactual reasoning. However, these works mostly focus on analyzing the performance of LLMs, and there has been comparatively little work in evaluating the causal reasoning capability of large visio-linguistic systems. Leveraging breakthroughs in contrastive language-image pretraining approaches such as CLIP Radford et al. 2021 , LVLMs have sparked new research questions about the capabilities of language models augmented with visual information. Often constructed by integrating a CLIP-like vision encoder in the LLM architecture and aligning the two modalities via projection, LVLMs have shown tremendous potential in tasks such as recognition, grounding, and VQA Li et al. 2025 . Recently, Zong et al. 2025 benchmarked LVLMs on diverse tasks with multi-modal in-context learning (ICL), and ongoing efforts are being made to incorporate and improve reasoning in models such as DeepSeek R1 Guo et al. 2025 and OpenAI o3. Similar to causal evaluations on LLMs, it is important to examine the causal reasoning ability of LVLMs, as building language and vision agents capable of reasoning and planning is paramount to the reliable usage of LLMs and LVLMs in real-world scenarios Gkountouras et al. 2025 . However, it is significantly more challenging for AI systems to learn causal relationships from high-dimensional data such as images that consist of complex causal dependencies in physical and dynamical systems Scholkopf et al. 2021 ; Komanduri et al. 2024 . In this work, we investigate the causal reasoning capabilities of LVLMs. Previously, Chen et al. 2024 evaluated the performance of LVLMs on causally-motivated VQA tasks where the causal relationships were defined in terms of observable interactions extracted from the scene graph representation of images. However, their evaluation focused on scene-specific relations originating from human-object interactions. In contrast, we focus on the ability of LVLMs to perform formal visual causal reasoning with systems described by deterministic causal mechanisms (e.g., light position causes a change in shadow length in Fig. 2(a) ). We formulate the visual causal reasoning task as the ability of LVLMs to disentangle causal variables and reason about their relationships, a fundamental goal of causal representation learning Scholkopf et al. 2021 . We further evaluate visual causal reasoning capabilities in LVLMs under zero-shot and few-shot learning settings. Assessing the causal reasoning capability of LVLMs in this setting can have significant implications for building robust AI systems in diverse domains. Our contributions are: (1) We formulate causal reasoning in LVLMs as inferring causal mechanisms from visual cues. (2) We construct a benchmark, CausalVLBench , encompassing three representative tasks: causal structure inference , intervention target prediction , and counterfactual prediction to evaluate the visual causal reasoning capabilities of LVLMs under zero and few shot settings. (3) We study the effect of prompting without the causal graph, demonstration selection, and zero-shot chain-of-thought on the performance of LVLMs. (4) We conduct rigorous empirical evaluations on several LVLMs to assess their performance in understanding the underlying causal mechanisms in physical systems. We primarily experiment with open-source models to maintain transparency and reproducibility. Our experiments indicate that formal causal reasoning while incorporating text and image modalities is challenging for current state-of-the-art LVLMs. We hope our evaluation elucidates the shortcomings of existing models and motivates the development of new training paradigms to promote causal reasoning in large multi-modal models Vashishtha et al. 2024 ; Rajendran et al. 2024 . 2 Preliminaries Causal Inference. Our work primarily relies on the causality framework by Pearl 2009 that mathematically formalizes the reasoning of cause and effect. The framework consists of a three-level hierarchy of causal inference referred to as the Ladder of Causation Pearl and Mackenzie 2018 or Pearl’s Causal Hierarchy (PCH) ( Bareinboim et al. 2022 ) . • Rung 1 , or \mathcal{L}_{1} , refers to statistical associations among random variables and involves reasoning about joint and conditional distributions. This rung describes questions such as “How often should I take medication when I am sick?” • Rung 2 , or \mathcal{L}_{2} , enables interventions on variables to reason about their effects. We can perform an intervention on a random variable X via the do operator (i.e., do (X=x) ). This rung describes questions such as “If I take the medication, will my sickness be cured?” • Rung 3 , or \mathcal{L}_{3} , deals with counterfactual reasoning (i.e., “what if?” questions) to imagine alternative scenarios in which the world could have been different. This rung describes questions such as “Would my sickness have been cured if I had taken the medication?” Reasoning about all of these quantities together requires the Structural Causal Model (SCM) formalism from Pearl 2009 as defined below. Definition 1 A structural causal model (SCM) is formally defined by a tuple \langle Z,U,F\rangle , where Z is the set of n endogenous variables, U is the set of n exogenous noise variables, and F is a set of structural equations of the form Z_{i}=f_{i}(Z_{pa_{i}},U_{i}) , where Z_{pa_{i}} are Z_{i} ’s causal parents. The conditionals P(Z_{i}|Z_{pa_{i}}) define the conditional distribution of Z_{i} given its parents. The joint observational distribution can then be factorized as follows: P(Z_{1},\dots,Z_{n})=\prod_{i=1}^{n}P(Z_{i}|Z_{pa_{i}}) (1) The observational (\mathcal{L}_{1}) , interventional (\mathcal{L}_{2}) , and counterfactual (\mathcal{L}_{3}) distributions entailed by the SCM form a hierarchy in the sense that \mathcal{L}_{1}\subset\mathcal{L}_{2}\subset\mathcal{L}_{3} , where each level encodes richer information that the previous level cannot express. In-Context Learning. Given a pretrained large vision-language model M_{\theta} , a text instruction I , and some support set S=(X_{i},Y_{i}) of query examples X along with the