Causal Debiasing for Visual Commonsense Reasoning
Jiayi Zou, Gengyun Jia, Bing-Kun Bao* · 2025
Read Causal Debiasing for Visual Commonsense Reasoning by Jiayi Zou, Gengyun Jia, Bing-Kun Bao* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20281.txt.
Introduction
Jiayi Zou, Gengyun Jia, and Bing-Kun Bao tackle bias in Visual Commonsense Reasoning, where a model answers questions about an image and explains its answer. High accuracy on standard datasets can hide reliance on co-occurrence and statistical biases in both the text and the images. The authors expose those biases and build two out-of-distribution test sets to measure true generalization across both modalities.
Drawing causal graphs of the task, they pinpoint prediction shortcuts that let models answer without real reasoning. A backdoor adjustment method then removes the bias, using a dictionary built from the set of correct answers to break the shortcut links. Experiments across different datasets show the debiasing method works, giving a clearer picture of what the models actually understand.
Catalog record
| CREATOR | Jiayi Zou, Gengyun Jia, Bing-Kun Bao* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20281
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Causal Debiasing for Visual Commonsense Reasoning Thanks: *corresponding author. This work was supported by the National Natural Science Foundation of China under Grants (No.62325206, 62306150), and the Key Research and Development Program of Jiangsu Province under Grant BE2023016-4. Jiayi Zou, Gengyun Jia, Bing-Kun Bao* Abstract Visual Commonsense Reasoning (VCR) refers to answering questions and providing explanations based on images. While existing methods achieve high prediction accuracy, they often overlook bias in datasets and lack debiasing strategies. In this paper, our analysis reveals co-occurrence and statistical biases in both textual and visual data. We introduce the VCR-OOD datasets, comprising VCR-OOD-QA and VCR-OOD-VA subsets, which are designed to evaluate the generalization capabilities of models across two modalities. Furthermore, we analyze the causal graphs and prediction shortcuts in VCR and adopt a backdoor adjustment method to remove bias. Specifically, we create a dictionary based on the set of correct answers to eliminate prediction shortcuts. Experiments demonstrate the effectiveness of our debiasing method across different datasets. I Introduction The Visual Commonsense Reasoning (VCR) [ 1 ] task is an extension of Visual Question Answering (VQA) [ 2 , 3 ] and consists of two sub-tasks: four-choice question answering (Q \rightarrow A) and reasoning (QA \rightarrow R). In the Q \rightarrow A sub-task, the model selects the correct answer from four options based on the image. In the QA \rightarrow R sub-task, the model selects the appropriate reasoning for the chosen answer. Existing research, including task-specific [ 4 , 5 , 6 , 7 , 8 ] and pre-training models [ 9 , 10 , 11 , 12 , 13 , 14 ] , has made a lot of progress in VCR. Task-specific models use attention mechanisms [ 15 , 16 , 17 ] or graph networks [ 18 , 19 , 20 ] to capture image and question correlations for accurate answering and reasoning. Pre-trained models leverage prior training on other datasets and fine-tuning on VCR, benefiting from learned relationships and enhancing visual-textual alignment [ 21 , 22 ] . Despite significant advancements in VCR, researchers have found that models may rely on biases rather than reasoning capabilities to answer questions [ 7 , 23 ] . The issue is initially investigated by Ye et al. [ 24 ] , who find that models heavily depend on co-occurrence for predictions when encountering repeated references. However, their study focus solely on the co-occurrence of person tags and do not extensively analyze other types of bias. Subsequent research has not adequately addressed the challenge of bias mitigation [ 25 ] . To bridge this gap, our study investigates bias challenges in VCR, focusing on co-occurrence bias and statistical bias. Co-occurrence bias arises from co-occurring components between modalities. During multi-modal feature fusion, the model may pay more attention to the common words between questions and answers or the co-occurrence between visual objects and answers, leading to biased predictions. To validate this, we conduct experiments using the R2C [ 1 ] on segmented datasets with or without co-occurring words between questions and answers. The results in Table I confirm that the baseline model achieves higher accuracy on the co-occurrence split, supporting our hypothesis. Moreover, we observe that certain question types and image types can introduce statistical bias. For example, when questions include specific verbs, the frequency of words in the ground-truth answers may vary. In such cases, the model may rely excessively on high-frequency words and undermine visual information. To investigate this, we divide the dataset based on the frequency of words in answers for each question type. This results in two subsets: one with high-frequency words (head-labeled) and another with low-frequency (tail-labeled). Experiments in Table I show that accuracy of Q \rightarrow A is higher on the head-labeled dataset, providing evidence for statistical bias. TABLE I: The R2C model predicts accuracy(%) on different split data sets to verify the existence of two biases. Subsets Q \rightarrow A QA \rightarrow R co-occurrence 36.9 44.4 None co-occurrence 31.3 30.9 head-labeled 29.6 – tail-labeled 27.9 – After analyzing the bias, we introduce the out-of-distribution (OOD) VCR-OOD dataset, which deals with both co-occurrence bias and statistical bias. For the text modality, we create VCR-OOD-QA by filtering samples with co-occurring words between questions and answers, and balancing verb frequency in answers for different question types. For the visual modality, we apply similar procedures to obtain VCR-OOD-VA. To combat bias, we develop a debiasing model based on causal intervention [ 26 , 27 ] . Given the causal graph, we depict the causal relationships in VCR and analyze the prediction shortcuts provided by confounding factors that lead to model learning bias. By employing the backdoor adjustment method, we design an answer dictionary to disrupt the propagation path of these confounding factors and eliminate bias. The backdoor adjustment approach allows us to adjust for the effect of confounding factors by conditioning on variables that block the spurious associations between the input and the output. We create the dictionary based on the correct answer set, ensuring that the model focuses on the relevant information rather than relying on shortcuts that might arise due to biases in the training data. In summary, our contributions can be categorized into three main areas: • We analyze two types of biases in VCR, including co-occurrence bias and statistical bias, and design VCR-OOD datasets. • In VCR, we utilize causal graphs to capture the influence of confounding factors on predictions. • To cut off the shortcut and alleviate bias, we introduce a debiasing method based on the backdoor adjustment to correct multi-modal features. II Construction of OOD datasets In this section, we leverage the in-distribution (ID) dataset VCR [ 1 ] to create the VCR-OOD dataset, which comprises two subsets: VCR-OOD-QA for the text modality and VCR-OOD-VA for the visual modality. Below we will introduce our processing method in detail. For the text modality, the VCR-OOD-QA dataset is constructed with the focus on co-occurrence bias and statistical bias. We filter out samples where there are co-occurring words between the questions and the correct answers, as well as between the connected question-answer pairs and the correct reasoning. This process create an auxiliary dataset. To tackle statistical bias, we first categorize the questions based on the verbs they contain, forming different question sets ( Q_{1} , Q_{2} , etc.). We then record the corresponding correct answer sets ( A_{1} , A_{2} , etc.) and extract the verb or noun from them. Based on the frequency of these words, we divide the samples for each question type into ”head-labeled” (high-frequency) and ”tail-labeled” (low-frequency) subsets. The division threshold, denoted as \alpha , corresponds to the frequency of the word ranked in the middle. Finally, we randomly retain a threshold number of ”head-labeled” samples from each question type to create the VCR-OOD-QA dataset. To maintain dataset consistency, we randomly select 10,000 training samples to balance the 1,045 validation samples. In the visual modality, establishing correspondence between objects and answer text can be challenging. However, as many questions describe objects present in the images, we leverage the auxiliary dataset in VCR-OOD-QA to mitigate this bias. We generate VCR-OOD-VA using a similar methodology, but categorize images based on the contained objects rather than question text. This results