WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata

Basel Shbita, Pengyuan Li, Anna Lisa Gentile · 2026

Read WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata by Basel Shbita, Pengyuan Li, Anna Lisa Gentile (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21479.txt.

Introduction

Basel Shbita, Pengyuan Li, and Anna Lisa Gentile introduce WikiVQABench, a knowledge-grounded visual question answering benchmark from Wikipedia and Wikidata. Most visual question answering benchmarks stress sight-based tasks solvable from visual content alone, yet many real cases require outside knowledge not seen in the image. The benchmark is human-curated, built by combining site images, article captions, and structured knowledge. Draft sets generated by language models are reviewed by human checks.

Catalog record

CREATORBasel Shbita, Pengyuan Li, Anna Lisa Gentile
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36073
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21479

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WikiVQABench: A Knowledge-Grounded Visual Question Answering Benchmark from Wikipedia and Wikidata Basel Shbita, Pengyuan Li, Anna Lisa Gentile Abstract Visual Question Answering (VQA) benchmarks have largely emphasized perception-based tasks that can be solved from visual content alone. In contrast, many real-world scenarios require external knowledge that is not directly observable in the image to answer correctly. We introduce WikiVQABench, a human-curated knowledge-grounded VQA benchmark constructed by systematically combining Wikipedia images, their associated article captions, and structured knowledge from Wikidata. Our pipeline uses large language models (LLMs) to generate candidate multiple-choice image-question-answer sets. All generated instances are subsequently reviewed and curated by human annotators to ensure factual correctness, visual-text consistency, and that each question requires external knowledge in addition to visual evidence for correct resolution. WikiVQABench comprises a substantial collection of Wikipedia images with curated multiple-choice questions designed to benchmark knowledge-aware vision-language models (VLMs). Evaluation of fifteen VLMs (256M-90B parameters) reveals a wide performance range (24.7%-75.6% accuracy), demonstrating that the benchmark effectively discriminates model capabilities on knowledge-intensive reasoning. The dataset and benchmarking code are publicly available. 1 Introduction Multimodal large language models (MLLMs) and vision-language models (VLMs) have recently shown strong capabilities across tasks involving vision and language, including image captioning, visual question answering (VQA), document understanding, and chart interpretation ( 14 ; 3 ; 27 ; 18 ; 10 ; 22 ; 5 ; 6 ) . Despite this progress, evaluating whether such models can reason beyond visual perception remains challenging, largely due to limitations in existing datasets and benchmarks. The majority of VQA benchmarks currently used for evaluation are limited in scope. Popular datasets focus on synthetic scenes, natural images with short factual queries, or narrow domains such as scientific diagrams and charts ( 13 ; 24 ; 7 ; 30 ) . These benchmarks often emphasize surface-level perception, such as object recognition or scene description, leaving open the broader challenge of knowledge-grounded visual cognition, where answering a question requires not only looking at an image but also understanding its real-world context, entities, and relationships. Several datasets have attempted to incorporate external knowledge into VQA, including OK-VQA ( 16 ) , A-OKVQA ( 19 ) , and KVQA ( 20 ) . While these efforts represent important steps toward knowledge-aware evaluation, they exhibit notable limitations from a benchmarking perspective. Many questions cover restricted sets of entities or domains or use open-ended answer formats that complicate standardized and reproducible evaluation. More importantly, few existing benchmarks explicitly enforce that answering a question requires external knowledge beyond what can be inferred from the image itself, nor do they provide verifiable ground truth grounded in structured, machine-readable knowledge sources. Real-world applications of VQA increasingly demand benchmarks that reflect these requirements. Questions about historical artifacts, landmarks, public figures, or events often require recognizing entities in images and reasoning over their properties, relationships, and broader context using external knowledge. Large, publicly available resources such as Wikipedia and Wikidata ( 25 ) provide complementary visual, textual, and structured information that can support the construction of benchmarks where visual content serves as an anchor for entity-centric reasoning and answers can be traced back to explicit, verifiable facts. To address this gap, we introduce WikiVQABench, a human-curated, knowledge-grounded visual question answering benchmark constructed by systematically combining Wikipedia images, associated article-image captions, and structured knowledge from Wikidata. WikiVQABench is designed as a dataset and a benchmark to evaluate whether VLMs and MMLMs can integrate visual evidence with external, structured knowledge, rather than relying on visual perception alone. Figure 1: Example from WikiVQABench illustrating a knowledge-grounded multiple-choice VQA instance. The image depicts a spider whose taxonomic classification cannot be determined from visual appearance alone. Correctly answering the question requires external biological knowledge linking visual cues to entity-level taxonomy (e.g., family or genus), demonstrating the benchmark’s emphasis on required knowledge beyond surface-level perception. The dataset construction process combines automated generation with rigorous human curation. Candidate multiple-choice image-question-answer sets are first generated using large language models (LLMs) conditioned on image captions and verbalized Wikidata triples. These candidates are then reviewed and curated by human annotators to ensure factual correctness, visual-text consistency, and that each question cannot be answered from the image alone. By grounding answers in structured knowledge and enforcing knowledge necessity through human curation, WikiVQABench provides verifiable ground truth suitable for standardized evaluation and comparative benchmarking. WikiVQABench comprises 344 images paired with curated multiple-choice questions spanning a diverse set of entities, relations, and domains. The benchmark is designed to support reproducible and scalable evaluation of knowledge-aware VLMs and MLLMs, enabling controlled analysis of entity-based and multi-hop “understanding” capabilities. Figure 1 illustrates a representative example from WikiVQABench, where answering the question requires taxonomic knowledge that cannot be inferred from visual appearance alone. We open source WikiVQABench 1 1 1 Code available as part of VLMEvalKit at: https://github.com/open-compass/VLMEvalKit and make the benchmark dataset publicly accessible 2 2 2 Dataset available at: https://huggingface.co/datasets/ibm-research/WikiVQABench in hopes of encouraging further work in developing more rigorous, fine-grained evaluation methodologies in this space. Our main contributions are as follows: • A new knowledge-grounded VQA benchmark leveraging Wikipedia and Wikidata that requires external, structured knowledge beyond visual perception to answer correctly. • A human-curated dataset construction methodology that ensures factual correctness, verifiable ground truth, and enforced knowledge necessity. • Comprehensive evaluation across multiple state-of-the-art models , demonstrating the benchmark’s effectiveness in assessing knowledge-grounded visual reasoning capabilities. • An open and accessible benchmarking resource , including dataset documentation, metadata, and tools. 2 Related work Visual Question Answering (VQA) has been studied extensively through a wide range of datasets and benchmarks. Early benchmarks such as VQA-v2 ( 8 ) and GQA ( 12 ) primarily focus on perception-based reasoning, emphasizing object recognition, spatial relations, and compositional visual understanding. While these datasets have driven progress in visual reasoning, they are largely solvable using image content alone and do not require access to external knowledge. To address this limitation, several benchmarks have introduced external knowledge into VQA. Benchmarks like OK-VQA ( 16 ) , A-OKVQA ( 19 ) , and KVQA ( 20 ) have explored entity-centric and knowledge-aware VQA, but exhibit limitations: knowledge requirements are often shallow or implicit, domain coverage is restricted, and open-ended answer formats complicate standardized and reproducible benchmarking. More recent work has explored large-scale, knowledge-intensive VQA grounded in encyclopedic information. Encyclopedic VQA ( 17 ) introduces a substantial collection of question-answer pairs (221k question-answer pairs and around 1M VQA samples) supported by evidence from a curated knowledge base derived from Wikipedia, demonstrating the importance of retrieval-augmented access to external knowledge. EchoSight ( 29 ) proposes a retrieval-augmented generation framework that retrieves relevant Wikipedia articles based on visual input to support encyclopedic question answering. While these approaches highlight the importance of external knowledge, they primarily focus on retrieval at inference time rather than on constructing benchmarks that explicitly enforce knowledge necessity and provide structured, verifiable ground truth. Several datasets leverage Wikipedia as a source of visual and semantic information. The Wikipedia-based Image Text (WIT) dataset ( 21 ) associates images with Wikipedia articles and captions, enabling large-scale image-text learning. OVEN ( 11 ) introduces open-domain visual entity recognition by unifying image classification and QA datasets under a shared label space grounded in English Wikipedia, covering a broad range of

Read the complete text