What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning

Zhaotian Weng, Haoxuan Li, Kuan-Hao Huang, Jieyu Zhao · 2025

Read What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning by Zhaotian Weng, Haoxuan Li, Kuan-Hao Huang, Jieyu Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00869.txt.

Introduction

Zhaotian Weng and co-authors probe whether vision-language models grasp causal order in a paper from two thousand twenty-five. These models read images and text together and ace many tests. But they may pass by naming objects and actions instead of reasoning about cause.

The team built two benchmarks that isolate cause-and-order questions. Models shone at recognition yet scored barely above chance on the causal tasks. Training data rarely spell out causal links, which the authors blame. Extra training with hard negative cases lifted causal scores while keeping general skill. They call the gap a key target for the field.

Catalog record

CREATORZhaotian Weng, Haoxuan Li, Kuan-Hao Huang, Jieyu Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-00869

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning Zhaotian Weng, Haoxuan Li, Kuan-Hao Huang, Jieyu Zhao Abstract Despite the impressive performance of vision-language models (VLMs) on downstream tasks, their ability to understand and reason about causal relationships in visual inputs remains unclear. Robust causal reasoning is fundamental to solving complex high-level reasoning tasks, yet existing benchmarks often include a mixture of reasoning questions, and VLMs can frequently exploit object recognition and activity identification as shortcuts to arrive at the correct answers, making it challenging to truly assess their causal reasoning abilities. To bridge this gap, we introduce VQA-Causal and VCR-Causal, two new benchmarks specifically designed to isolate and rigorously evaluate VLMs’ causal reasoning abilities. Our findings reveal that while VLMs excel in object and activity recognition, they perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. Further analysis suggests that this limitation stems from a severe lack of causal expressions in widely used training datasets, where causal relationships are rarely explicitly conveyed. We additionally explore fine-tuning strategies with hard negative cases, showing that targeted fine-tuning can improve model’s causal reasoning while maintaining generalization and downstream performance. Our study highlights a key gap in current VLMs and lays the groundwork for future work on causal understanding. 1 1 1 The dataset and code is avaiable at https://github.com/limenlp/CausalVLM 1 Introduction Pre-trained vision-language models have demonstrated impressive performance across a wide range of tasks, including visual question answering ( Antol et al. 2015 ; Li et al. 2019 ) , reasoning ( Zellers et al. 2019 ) , and object detection ( Li et al. 2022b ) . However, strong performance on these benchmarks does not necessarily reflect a rich understanding of visual inputs. Recent studies have revealed that VLMs struggle with tasks demanding high-level visual understanding, such as verb comprehension, spatial reasoning, attribute attachment, and counting ( Wang et al. 2023 ; Kamath et al. 2023 ; Yuksekgonul et al. 2023 ; Chen et al. 2024 ; Parcalabescu et al. 2021 ) . Crucially, whether VLMs possess genuine causal reasoning abilities remains largely unexplored. For instance, can VLMs distinguish between “The woman holding an umbrella is caused by the rain.” and “The rain is caused by the woman holding an umbrella.”? Robust causal understanding and reasoning are fundamental to tackling complex real-world decision making ( Lake et al. 2017 ) , but this capability in VLMs remains largely unexplored. Existing benchmarks that aim to assess reasoning in VLMs often conflate causal reasoning with other types of reasoning tasks ( Antol et al. 2015 ; Zellers et al. 2019 ) , and many questions can be answered by object recognition or activity understanding alone. For example, our analysis of the Visual Question Answering (VQA) and Visual Commonsense Reasoning (VCR) benchmarks reveals that only 0.92% of questions in the VQA validation set ( Antol et al. 2015 ) and 35.43% in the VCR validation set ( Zellers et al. 2019 ) involve causal reasoning. Our analysis of 100 randomly selected VCR questions found that 46% could be answered correctly through object detection or activity understanding alone, without requiring genuine causal reasoning. These issues make it difficult for current benchmarks to independently and effectively evaluate the causal reasoning ability of VLMs. To address this gap, we introduce VQA-Causal and VCR-Causal, the first benchmarks specifically designed to rigorously and independently evaluate VLMs’ causal reasoning abilities. Constructed from different sources, VQA ( Antol et al. 2015 ) and VCR ( Zellers et al. 2019 ) , this dual-benchmark setup enables fine-tuning on VQA data and both in-domain (VQA-Causal) and out-of-domain (VCR-Causal) evaluation, thereby providing a robust assessment of models’ causal reasoning capabilities and their generalizability across datasets. VQA-Causal consists of 1,947 instances, and VCR-Causal contains 3,511 instances, with each image paired with 12 caption pairs using different causal conjunctions. Each caption pair differs only in the causal relationship between events, as illustrated in Figure 1 . This counterfactual approach ensures a comprehensive evaluation of the model’s understanding of causal relationships, avoiding potential biases toward specific causal expressions. Figure 1: Examples from the VQA-Causal test and the Object and Activity Understanding test. Models tend to focus on low-level visual features such as objects and activities which are represented by the red and blue nodes in the scene graph on the left, but fail to capture high-level visual features such as relationships between activities, especially causal relationships in our case, which are represented by the green nodes in the scene graph. We evaluate 10 widely-used VLMs, covering a broad spectrum of architectures and objectives, including score-based and generative models trained with diverse objectives. The evaluated models include CLIP ( Radford et al. 2021 ) , NegCLIP ( Yuksekgonul et al. 2023 ) , BLIP ( Li et al. 2022a ) , FLAVA ( Singh et al. 2022 ) , LLaVA ( Liu et al. 2023 ) and so on. All models perform poorly on both VQA-Causal and VCR-Causal, with nine out of ten achieving no more than 52% accuracy, which is only marginally above a random guess (50%) and significantly below human performance (98%). These findings highlight a fundamental limitation of existing VLMs in causal reasoning. To better understand whether the poor performance on causal reasoning tasks stems from a lack of basic visual understanding, we constructed a controlled evaluation set by modifying the VQA-Causal dataset. This modified dataset contains the same 1,947 instances as VQA-Causal, but each image is paired with four captions, two of which correctly describe the image. The incorrect captions differ by altering the object or modifying the described activity, as illustrated in Figure 1 . Our results reveal that while VLMs perform well in recognizing objects and activities, they struggle significantly with reasoning about causal relationships between activities, further reinforcing our findings from VQA-Causal and VCR-Causal. We then investigate why VLMs trained on large-scale image-text corpora fail to learn causal relationships between events in visual inputs. Focusing on LAION-400M ( Schuhmann et al. 2021 ) (used by OpenCLIP) and MSCOCO ( Lin et al. 2014 ) (used in FLAVA and NegCLIP) ( Singh et al. 2022 ; Yuksekgonul et al. 2023 ) , we found that explicit causal expressions are extremely rare. Quantitatively, only 0.08% of LAION-400M and 0.01% of MSCOCO instances contain explicit causal expressions. This scarcity explains why VLMs excel at object and activity recognition but struggle with causal reasoning. To mitigate this limitation, we explored fine-tuning strategies incorporating hard negative cases, captions that differ from the correct ones only in the causal order, demonstrating that targeted fine-tuning can significantly enhance causal reasoning. Our fine-tuned model, CausalCLIP, achieves notable improvements on both in-domain and out-of-domain benchmarks while maintaining strong performance on downstream tasks. Our contributions are as follows: • We introduce VQA-Causal and VCR-Causal, the first benchmarks specifically designed to isolate and comprehensively evaluate causal reasoning in VLMs, addressing a critical limitation in existing benchmarks. Moreover, this setup allows us to use one dataset (e.g., VQA) as an in-domain source for fine-tuning and evaluate the model’s causal reasoning ability on both in-domain (VQA-Causal) and out-of-domain (VCR-Causal) benchmarks to assess generalization. • Our experimental results reveal that while VLMs excel in object and activity understanding, they

Read the complete text