Are a Thousand Words Better Than a Single Picture? Beyond Images - A Framework for Multi-Modal Knowledge Graph Dataset Enrichment

Pengyu Zhang, Klim Zaporojets, Jie Liu, Jia-Hong Huang, Paul Groth · 2026

Read Are a Thousand Words Better Than a Single Picture? Beyond Images - A Framework for Multi-Modal Knowledge Graph Dataset Enrichment by Pengyu Zhang, Klim Zaporojets, Jie Liu, Jia-Hong Huang, Paul Groth (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16974.txt.

Introduction

Zhang and co-authors enrich multimodal knowledge graphs by turning images into text. Large image collections are hard to curate, and ambiguous visuals like logos are often dropped. Their pipeline, Beyond Images, retrieves more entity related images, converts all visuals into text descriptions, and fuses those texts into short entity summaries. These summaries can replace or augment the text side of standard models. Across three public datasets, the authors report gains up to 7 percent in Hits@1. On ambiguous logos, gains were far larger.

Catalog record

CREATORPengyu Zhang, Klim Zaporojets, Jie Liu, Jia-Hong Huang, Paul Groth
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43343
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16974

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Are a Thousand Words Better Than a Single Picture? Beyond Images - A Framework for Multi-Modal Knowledge Graph Dataset Enrichment Pengyu Zhang, Klim Zaporojets, Jie Liu, Jia-Hong Huang, Paul Groth Abstract Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images , an automatic data-centric enrichment pipeline with optional human auditing. This pipeline operates in three stages: (1) large-scale retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG datasets and multiple baseline models, we observe consistent gains (up to +7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements ( +201.35% MRR and +333.33% Hits@1). Additionally, we release a lightweight Text–Image Consistency Check Interface for optional targeted audits, improving description quality and dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code, datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images . 1 Introduction In Multi-Modal Knowledge Graphs (MMKGs), integrating text, images, audio, and video with structured triples yields richer signals and improves entity representations [ 5 , 12 , 24 ] . As illustrated on the left of Figure 1 , the entity “ Amsterdam ” may be associated with textual descriptions of its history and culture, photographs of canals and landmarks, and audio or visual content that reflects its urban atmosphere. Among these modalities, images often convey dense information, and when their content is unambiguous, downstream performance improves [ 22 ] . However, even though the Web enables rapid, large-scale collection of entity-related images, the construction process still suffers from two limitations. Figure 1: Original MMKG Dataset : the entity “ Amsterdam ” with its photo and a textual description. Ambiguous yet Relevant Images : additional visuals such as the three red Saint Andrew’s crosses and an abstract cityscape, whose semantics may be unclear when incorporated directly as visual features. Textual Descriptions of the Additional Ambiguous Images : our framework converts these images into concise, entity-aligned text, expanding semantic coverage while mitigating noise from ambiguous visual embeddings. Limitation 1: Lack of scalability. While most existing MMKG datasets automate the acquisition of textual descriptions, image collection often still relies on manual curation [ 31 ] . Curators or domain experts typically search for each entity and select a small set of images [ 20 ] . This practice helps avoid obviously misleading content but requires manual collection or verification and is difficult to scale to large graphs [ 17 , 2 ] . Limitation 2: Ambiguous yet relevant images remain unused. Manual curation typically favors highly representative images for each entity and excludes uncertain or ambiguous images [ 3 , 7 ] . Although this strategy supports quality control, it narrows the dataset’s visual coverage and suppresses alternative yet valid information. The Web contains a large number of entity-related images whose visual semantics are inherently ambiguous. When incorporated directly into a dataset, such images often contribute noise rather than useful signal, which lowers data quality and degrades model performance [ 6 , 11 ] . Figure 1 highlights two common forms of image ambiguity: Sparse-Semantic Images (e.g., symbolic logos) are visually simple and lack sufficient semantic detail. Although sometimes domain-relevant, they rarely provide distinctive embeddings that benefit MMKG models [ 26 , 28 ] . In contrast, Rich-Semantic Images (e.g., abstract paintings) contain visually and semantically complex scenes that current embedding methods struggle to interpret, often leading to significant information loss [ 29 ] . Consequently, such images are frequently discarded during dataset construction. While this preserves highly reliable visual signals, it also reduces within-entity diversity. For example, in Figure 1 , for the entity “ Amsterdam ”, a dataset may only retain canal photographs but exclude the “triple red X” despite its historical and cultural significance to the city. To address these limitations, we propose an automated framework, Beyond Images , which comprises three stages. First, the Modality Extension Module retrieves additional entity-related images beyond the original dataset, enabling large-scale expansion. Next, the Semantic Generation Module converts all retrieved images into textual descriptions, ensuring that ambiguous visuals contribute usable semantics rather than noise. Finally, the LLM-based Semantic Fusion Module consolidates multi-source descriptions, filters task-irrelevant content, and produces concise, entity-aligned summaries. Our experiments show that our framework effectively addresses the two limitations defined above. In terms of scalability, we observe consistent improvements across three public MMKG datasets, achieving gains of up to +7% in Hits@1. To evaluate the impact of ambiguous yet relevant images (e.g., logo-only or symbol-centric visuals), we retain and convert newly retrieved images and measure link-prediction gains from their descriptions. On this challenging subset, performance increases by +201.35% MRR and +333.33% Hits@1. Additionally, we assess the quality of the generated image descriptions and show that our fused textual summaries outperform purely generated summaries in terms of overall quality. To support this evaluation, we implement a Text-Image Consistency Check Interface that allows for efficient targeted quality assessments by human annotators. Our main contributions are as follows: (i) We propose Beyond Images , a reusable data-centric enrichment paradigm for MMKGs that converts ambiguous-yet-relevant visuals (e.g., logos or symbolic images) into entity-aligned textual representations and performs entity-level semantic fusion, without modifying downstream model architectures or training objectives. (ii) We develop a lightweight Text–Image Consistency Check Interface to support optional human auditing, enabling low-effort quality inspection of generated entity summaries. (iii) We release enriched datasets, code, and evaluation protocols, and conduct a systematic study across three datasets on four MMKG models, demonstrating consistent gains, up to +7% in Hits@1 overall, and +333.33% in Hits@1 on a challenging subset. 2 Related Work Multi-Modal Knowledge Graphs. Link prediction is a core task in Knowledge Graphs (KGs), aiming to infer missing entities or relations from existing triples. Multi-Modal Knowledge Graphs (MMKGs) extend this setting by combining data from multiple modalities, such as text, images, and numerical features, to enhance representation learning and prediction accuracy. [ 20 ] introduced MMKGs that integrate numerical and visual information, demonstrating improvements in link prediction. Building on this, MCLEA [ 19 ] proposed a contrastive learning framework for multi-modal entity alignment. MCLEA first learns modality-specific representations and then applies contrastive learning to jointly model intra-modal and inter-modal interactions. Subsequently, MMKRL [ 21 ] incorporated a knowledge reconstruction module to integrate structured and multi-modal data into a unified space. This model also uses adversarial training to enhance robustness and performance. More recently, [ 14 ] introduced the MR-MKG method, which leverages MMKGs to improve reasoning capabilities in large language models. In parallel, [ 4 ] developed the SNAG model, which effectively combines structural, visual, and textual features, yielding improved link prediction performance. [ 35 ] leverages a multi-modal knowledge graph built from news text and

Read the complete text