MKG-RAG-Bench : Benchmarking Retrieval in Multimodal Knowledge Graph–Augmented Generation Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3817570 ISBN: 979-8-4007-2259-2/2026/08 CCS: Computing methodologies Continuous space search CCS: Computing methodologies Natural language generation
Xiaochen Wang, Bao Hoang, Han Liu, Ting Wang, Fenglong Ma · 2026
Read MKG-RAG-Bench : Benchmarking Retrieval in Multimodal Knowledge Graph–Augmented Generation Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3817570 ISBN: 979-8-4007-2259-2/2026/08 CCS: Computing methodologies Continuous space search CCS: Computing methodologies Natural language generation by Xiaochen Wang, Bao Hoang, Han Liu, Ting Wang, Fenglong Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26458.txt.
Introduction
Xiaochen Wang, Bao Hoang, Han Liu, Ting Wang, and Fenglong Ma built MKG-RAG-Bench, a benchmark for retrieval in multimodal knowledge graph systems. Such systems answer questions by fetching facts from a knowledge graph, a network of linked facts. Multimodal graphs mix text with images and other media. Retrieval, the fetching step, is often the weak link.
Existing benchmarks mostly ignore that retrieval problem. The new benchmark draws on two multimodal knowledge graphs. One covers general topics and the other covers medicine. Aligned question-and-answer sets support controlled tests of retrieval and of final answer quality. A language model pipeline built the benchmark. It filters low-value knowledge and writes questions with exact known answers. It also covers varied mixes of media types. The authors use it to compare retrievers and show where multimodal retrieval breaks down.
Catalog record
| CREATOR | Xiaochen Wang, Bao Hoang, Han Liu, Ting Wang, Fenglong Ma |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26458
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MKG-RAG-Bench : Benchmarking Retrieval in Multimodal Knowledge Graph–Augmented Generation Conference: Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2; August 09–13, 2026; Jeju Island, Republic of Korea Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD ’26), August 09–13, 2026, Jeju Island, Republic of Korea DOI: 10.1145/3770855.3817570 ISBN: 979-8-4007-2259-2/2026/08 CCS: Computing methodologies Continuous space search CCS: Computing methodologies Natural language generation Xiaochen Wang, Bao Hoang, Han Liu, Ting Wang, Fenglong Ma Abstract . Retrieval-augmented generation (RAG) over knowledge graphs has emerged as a promising approach for grounding large language models, yet existing benchmarks largely overlook the challenges of retrieval in multimodal knowledge graph RAG (MKG-RAG). In practice, retrieval is a critical bottleneck: multimodal knowledge is heterogeneous, difficult to align across modalities, and often poorly served by retrievers designed for unstructured corpora. To address this gap, we introduce MKG-RAG-Bench , a cross-domain benchmark explicitly designed to evaluate retrieval in MKG-RAG. MKG-RAG-Bench is constructed from two multimodal knowledge graphs spanning general and medical domains, and includes carefully aligned question-answering datasets that support controlled evaluation of both retrieval and downstream generation. The benchmark is built using an LLM-based curation pipeline that filters low-utility knowledge, generates structurally grounded queries with exact supervision, and systematically covers diverse modality configurations. Through extensive experiments across representative retriever families and modality settings, we show that effective multimodal retrieval remains challenging yet crucial for end-to-end MKG-RAG performance, and that retrieval quality strongly determines generation outcomes. By isolating retrieval as a first-class evaluation target, MKG-RAG-Bench provides a principled foundation for diagnosing current limitations and advancing multimodal knowledge graph RAG systems. 1 1 1 The benchmark can be accessed via: https://github.com/XiaochenWang-PSU/MKG-RAG-Bench . † † cc-license: by 1. Introduction Retrieval-augmented generation (RAG) enhances large language models (LLMs) by retrieving information relevant to a given query from external sources and conditioning generation on this retrieved evidence ( 16 ) . While RAG has proven effective in improving factual grounding, conventional approaches predominantly retrieve from unstructured textual corpora that are noisy, fragmented, and weakly connected. These limitations often hinder reliable evidence selection and multi-step reasoning, especially for complex queries. To address these issues, knowledge graph–based RAG (KG-RAG) has been proposed ( 8 ) . By retrieving information from structured knowledge graphs, KG-RAG enables more coherent, contextualized, and comprehensive knowledge access, leading to improved answer quality and faithfulness ( 38 ) . However, existing KG-RAG methods are largely confined to textual knowledge graphs , which substantially limits their applicability in real-world scenarios where critical information is conveyed through images, charts, tables, and other non-textual modalities. Figure 1 . The proposed pipeline for benchmark construction using multimodal knowledge graphs and LLMs. Multimodal KG-RAG. To overcome this limitation, we explore the task, multimodal KG-RAG ( MKG-RAG ), which integrates structured knowledge across multiple modalities to support more expressive retrieval and reasoning. Formally, given a query q , a multimodal knowledge graph \mathcal{G} , a retriever Retriever , and a multimodal large language model MLLM , the goal of MKG-RAG is to generate a response \mathcal{R} , i.e., \mathcal{R}=\textsf{MLLM}(q,\textsf{Retriever}(q,\mathcal{G})). The most closely related setting is multimodal RAG ( 3 ) , which follows the traditional RAG pipeline by retrieving multimodal content from unstructured external corpora (e.g., Wikipedia) rather than from a structured multimodal knowledge graph. To the best of our knowledge, no existing benchmark or prior work systematically studies retrieval and generation in the MKG-RAG setting. Motivation and Challenges. Multimodal KG-RAG is benchmark-worthy because it evaluates a model’s ability to retrieve, align, and reason over structured multimodal knowledge , a critical capability for real-world systems that is not captured by existing RAG, KG-RAG, or multimodal benchmarks. This work aims to introduce the first benchmark for the MKG-RAG task, consisting of curated datasets and evaluation protocols that explicitly target retrieval quality and multimodal grounding. Designing such a benchmark, however, poses several non-trivial challenges: • Heterogeneous Retrieval. Unlike conventional knowledge graphs, where both queries and target triplets are unimodal, multimodal knowledge graphs support heterogeneous modality combinations across queries and retrieval targets. In MKG-RAG , queries may be text-only, image-only, or multimodal, while relevant graph components may combine textual, visual, and numerical information in different ways. These combinations induce distinct retrieval behaviors and demand different modeling strategies. A meaningful MKG-RAG benchmark must therefore systematically cover diverse multimodal retrieval scenarios to reflect realistic usage and to enable fair comparison among retrievers. • Multimodal Knowledge Graph–Query Alignment. In MKG-RAG , relevant knowledge may not be explicitly aligned with the query at the surface level, particularly when the query and target nodes differ in modality or abstraction. For example, a textual query may require grounding in visual evidence, while an image-based query may necessitate retrieving semantically related textual entities and relations. Addressing such cross-modal and cross-structural alignment requires joint reasoning over graph topology, semantic relationships, and modality-specific representations. Accordingly, a MKG-RAG benchmark must include queries that demand non-trivial alignment beyond shallow modality matching. • Benchmark Utility. The primary goal of the benchmark is to evaluate both the effectiveness of multimodal retrieval and the grounding quality of multimodal LLMs. This requires careful construction to ensure that the retrieved knowledge is informative and well-aligned with downstream queries. Multimodal knowledge graphs often contain generic or weakly informative triplets, which can lead to ambiguous or poorly grounded questions and obscure the contribution of retrieval. A useful MKG-RAG benchmark must prioritize high-utility triplets and disentangle improvements due to effective retrieval from those arising from generative priors. Our Solution. To address these challenges, we propose a cross-domain, retrieval-oriented MKG-RAG benchmark, named MKG-RAG-Bench , comprising two multimodal knowledge graphs and corresponding multimodal question-answering datasets. The benchmark is designed to enable comprehensive evaluation across both the retrieval and generation stages of multimodal KG-RAG. The benchmark construction pipeline is shown in Figure 1 . Specifically, we use an LLM to select high-utility triplets and employ heuristic strategies to ensure strong alignment between the selected triplets and the constructed queries. The same LLM, with different prompts, is further leveraged to generate multimodal, knowledge-grounded questions across diverse retrieval scenarios, thereby systematically covering heterogeneous modality combinations. We conduct extensive experiments with representative retriever architectures across different multimodal settings, evaluating both effectiveness and efficiency throughout the retrieval and generation pipeline. While this work primarily focuses on evaluating MKG-RAG using training-free approaches, the benchmark is also applicable to training-based methods, including retriever training and MLLM fine-tuning. Table 1 . Comparison of representative multimodal RAG and KG-RAG benchmarks. Benchmark Name Knowledge Source Type Knowledge Structure Retrieval Unit Heterogeneous Query Support Evaluation Granularity Cross-Domain Coverage OK-VQA ( 26 ) Web Unstructured Passage Multimodal Generation-only Single-domain M 2 RAG ( 25 ) Web Unstructured Passage and Image Multimodal Generation-only Single-domain CRAG-MM ( 37 ) Web Ad-hoc MKG Passage and Image Multimodal Generation-only Multi-domain Dyn-VQA ( 20 ) Web Unstructured Passage and Image Multimodal Generation-only Multi-domain WebQSP ( 43 ) Textual KG Textual KG Textual Triplet Text-only Generation-only Single-domain CWQ ( 35 ) Textual KG Textual KG Textual Triplet Text-only Generation-only Single-domain GrailQA ( 10 ) Textual KG Textual KG Textual Triplet Text-only Generation-only Multi-domain REAL-MM-RAG ( 41 ) Multimodal Documents PDF Chunk Text-only Retrieval-only Multi-domain DocBench ( 46 ) Multimodal Documents PDF Chunk Text-only Generation-only Multi-domain MMLongBench ( 24 )