MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering
Shuo Yang, Soyeon Caren Han, Siwen Luo, Eduard Hovy · 2025
Read MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering by Shuo Yang, Soyeon Caren Han, Siwen Luo, Eduard Hovy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18491.txt.
Introduction
Shuo Yang and co-authors add commonsense knowledge to visual question answering in a 2025 paper. Answering questions about images needs facts about the everyday world, not just what pixels show, and large vision-language models often lack that grounding. Their framework, MAGIC-VQA, feeds commonsense into the model in three stages.
First, explicit knowledge is pulled from outside sources. Next, a by-type post-processing step refines it for the question's context. Finally, a graph neural network adds implicit knowledge through structured reasoning over linked facts. The design needs no heavy pre-training or complex prompt tuning. Tests on benchmark data sets reached state-of-the-art results, with clear gains on questions that demand commonsense.
Catalog record
| CREATOR | Shuo Yang, Soyeon Caren Han, Siwen Luo, Eduard Hovy |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18491
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MAGIC-VQA: Multimodal And Grounded Inference with Commonsense Knowledge for Visual Question Answering Shuo Yang, Soyeon Caren Han, Siwen Luo, Eduard Hovy Abstract Visual Question Answering (VQA) requires reasoning across visual and textual modalities, yet Large Vision-Language Models (LVLMs) often lack integrated commonsense knowledge, limiting their robustness in real-world scenarios. To address this, we introduce MAGIC-VQA, a novel framework that enhances VQA by systematically integrating commonsense knowledge with LVLMs. MAGIC-VQA employs a three-stage process: (1) Explicit Knowledge Integration from external sources, (2) By-Type Post-Processing for contextual refinement, and (3) Implicit Knowledge Augmentation using a Graph Neural Network (GNN) for structured reasoning. While GNNs bring greater depth to structured inference, they enable superior relational inference beyond LVLMs. MAGIC-VQA bridges a key gap by unifying commonsensse knowledge with LVLM-driven reasoning, eliminating the need for extensive pre-training or complex prompt tuning. Our framework achieves state-of-the-art performance on benchmark datasets, significantly improving commonsense reasoning in VQA. Our implementation is open-sourced on GitHub 1 1 1 https://github.com/adlnlp/magic_vqa 1 Introduction Visual Question Answering (VQA) Antol et al. (2015) ; Goyal et al. (2017) ; Yue et al. (2024) is a complex task requiring models to understand the interaction between visual inputs and textual queries. In recent years, Large Vision-Language Models (LVLMs) Dai et al. (2023) ; Wang et al. (2024b) ; Liu et al. (2024b) ; Li et al. (2024a) ; OpenAI (2024b) have made substantial progress in VQA through extensive pre-training on massive image-text datasets and instruction tuning. These models excel at object-level visual recognition and semantic understanding, capturing attributes such as spatial relationships and contextual details. Nevertheless, LVLMs often face challenges on questions requiring commonsense reasoning, particularly those hinging on implicit contextual cues or everyday world knowledge Zhou et al. (2023) ; Ye et al. (2022) ; Li et al. (2023b) 2 2 2 The sample illustrations can be found in Section 6 and Appendix B . . To overcome this limitation and improve the performance on , different methods have been explored. For example, Multimodal retrieval-augmented generation leverages dense retrieval to inject external multimodal information into the generation process, thereby enhancing the factual grounding of LVLMs Lin and Byrne (2022) ; Hu et al. (2023) . Multimodal prompt tuning harnesses the model’s innate commonsense knowledge by carefully crafting prompts that combine visual and textual cues from representative samples, guiding LVLMs to leverage their internal reasoning for context-rich answers Wei et al. (2022) ; Zhang et al. (2023) . However, static prompt design usually lacks the dynamic adaptability required for novel scenarios, resulting in limited generalization to unseen or diverse inputs. Additionally, graph-based approaches utilize Graph Neural Networks (GNNs) to incorporate structured commonsense knowledge Ravi et al. (2023) ; Wang et al. (2022) , which surpasses the limitations of purely parametric LVLMs, enabling models to capture explicit and implicit knowledge connections via structured graphs. However, a key missing component in existing works is the effective integration of commonsense knowledge with LVLMs while addressing their inherent shortcomings. Prior approaches either rely on static retrieval that indiscriminately injects input-unaware noisy knowledge or graph-based augmentation that overlooks the dynamic interplay between external and innate knowledge. Our work seeks to fill this gap by proposing a unified framework that systematically combines dynamic, contextually aligned commonsense integration with structured graph-based reasoning to robustly filter and incorporate relevant commonsense knowledge. In this paper, we introduce MAGIC-VQA, a novel framework designed to enhance VQA models by effectively integrating commonsense knowledge with LVLMs. MAGIC-VQA is built upon a three-stage process that not only improves reasoning capabilities but also mitigates the complexity associated with large-scale pre-training and inefficient prompt-based approaches. First, explicit Commonsense Knowledge Integration extracts relevant knowledge triples from external sources, establishing a reliable reasoning foundation. Secondly, by-Type Commonsense Knowledge Post-processing refines these triples based on input-specific needs, ensuring contextual relevance. Finally, implicit Commonsense Knowledge Augmentation constructs a heterogeneous multimodal graph processed by a GNN to capture intricate relationships, providing structured reasoning beyond what LVLMs alone can infer. By integrating explicit and implicit commonsense knowledge on top of LVLMs, MAGIC-VQA addresses both the limitations of previous approaches and the missing component in existing works. Our main contributions are as follows: 1. We propose MAGIC-VQA, a novel end-to-end framework that systematically integrates both explicit and implicit commonsense knowledge into VQA through, without extensive pre-training or intricate prompt tuning. 2. MAGIC-VQA employs a three-stage pipeline—explicit commonsense integration, by-type post-processing, and graph-based implicit augmentation-that dynamically extracts and filters commonsense knowledge in an input-aware manner, and leverages a GNN-based structured reasoning mechanism. 3. We conduct extensive evaluations across multiple VQA benchmarks, demonstrating robust improvement in commonsense understanding reasoning for VQA, surpassing existing models in both knowledge grounding and inference accuracy. 2 Related Work 2.1 VLPM and LVLMs on VQA Vision-Language Pretrained Models (VLPMs) like ViLBERT Su et al. (2019) , ALBEF Li et al. (2021) and VILT Kim et al. (2021) have advanced Visual Question Answering (VQA) by improving the alignment between visual and textual modalities in the last few years Long et al. (2022a) . Recently, Large Vision-Language Models (LVLMs) like InstructBLIP Dai et al. (2023) , LLaVA Liu et al. (2024b) , GPT4o OpenAI (2024b) and Gemini1.5 Gemini Team (2024) further push the boundary of VQA with strong in-context learning capability through extensive pre-training and instruction-tuning on large-scale image-text datasets. While these models have demonstrate their effectiveness in domains such as math Jiang et al. (2024) ; Wang et al. (2025a) , robotics Zhou et al. (2025a) , logical reasoning Bi et al. (2025b) ,medical Wang et al. (2025e) ; Wang et al. (2025d) and science Lu et al. (2022) , these models still face challenges with questions requiring commonsense knowledge that is intuitive and straightforward for humans, such as reasoning based on implicit contextual cues or general world knowledge Ye et al. (2023) ; Jiang et al. (2023) ; Chen et al. (2024a) ; Yang et al. (2025) ; Zhou et al. (2025b) ; Bi et al. (2025a) ; Jiang et al. (2025) . The resource-intensive nature of these models further makes it infeasible to train a model from scratch for enhanced commonsense understanding Bi et al. (2024) . 2.2 Commonsense Knowledge Integration for Visual Question Answering Several studies have highlighted the critical role of commonsense knowledge integration in enhancing the performance of VLPMs and LVLMs on VQA tasks Wu et al. (2022) ; Zhang et al. (2022) ; Ding et al. (2022) ; Wang et al. (2024c) ; Li et al. (2025a) ; Wang et al. (2025b) . These methods can be classified into two approaches: explicit commonsense knowledge integration and implicit commonsense knowledge integration. 1) Explicit commonsense knowledge integration directly incorporates external commonsense knowledge into model training through instruction tuning, prompt tuning, or reinforcement learning. For example, VLC-BERT Ravi et al. (2023) encodes the contextualized commonsense knowledge of the question phrases as additional textual features and integrates with object visual features to fine-tune the VL-BERT Su et al. (2019) . MM-CoT Zhang et al. (2023) , T-SciQ Wang et al. (2024a) and KAM-CoT Mondal et al. (2024) fine-tune models on commonsense-augmented Chain-of-Thought (CoT) data to enhance their reasoning processes. Prism Bi et al. (2025a) incorporate the filtered commonsense knowledge into model via dynamic data selection. Vl-Rethinker Wang et al. (2025b) and Pixel