Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models DOI: XXXXXXX.XXXXXXX Journal: JOCCH Volume: 37 4 111 8 CCS: Computing methodologies Artificial intelligence CCS: Information systems Information retrieval

Yang Zhang, Nada Mimouni, Jean-Claude Moissinac, Fayçal Hamdi · 2026

Read Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models DOI: XXXXXXX.XXXXXXX Journal: JOCCH Volume: 37 4 111 8 CCS: Computing methodologies Artificial intelligence CCS: Information systems Information retrieval by Yang Zhang, Nada Mimouni, Jean-Claude Moissinac, Fayçal Hamdi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17669.txt.

Introduction

Zhang and colleagues propose a multimodal approach for extending heritage graphs. They introduce WJoconde, a new French heritage knowledge graph joining text and images, plus three versions for downstream research, with a benchmark for completion methods. They also propose a framework for growing cultural graphs with large language and vision-language models, combining automated extraction with a validation pipeline. Results show joining text and image information well enhances graphs with high trust. All code, benchmark datasets and original data with a live access point are open-sourced.

Catalog record

CREATORYang Zhang, Nada Mimouni, Jean-Claude Moissinac, Fayçal Hamdi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56043
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Cultural Heritage Knowledge Graph Extension with Language and Vision Models DOI: XXXXXXX.XXXXXXX Journal: JOCCH Volume: 37 4 111 8 CCS: Computing methodologies Artificial intelligence CCS: Information systems Information retrieval Yang Zhang, Nada Mimouni, Jean-Claude Moissinac, Fayçal Hamdi Abstract . The preservation and interpretation of cultural heritage increasingly rely on digital technologies, among which Knowledge Graphs (KGs) stand out for their ability to structure vast amounts of data. However, the construction and expansion of these KGs often face challenges due to the diverse and complex nature of cultural heritage information. In this paper, we propose a novel approach for extending KG resources in the domain of cultural heritage, which we applied to French data. First, we introduce a new knowledge graph in the domain of French cultural heritage, WJoconde, which is distinguished by its multimodality as it integrates both textual and image information of the entities. We further introduce three variants of WJoconde to facilitate downstream research, such as Knowledge Graph Completion (KGC). We also built a comprehensive benchmark for KGC methods on our dataset. Second, we propose a new framework for extending cultural heritage KGs using multi-modal approaches leveraging Large Language Models (LLMs) and Vision-Language Models (VLMs), which includes automated data extraction from unstructured resources combined with a special validation pipeline for grounding the output of both models, to further extend WJoconde. Our results show that by integrating the rich text and image information in cultural heritage data, we can efficiently enhance KGs with high reliability. We open-source all code and benchmark datasets with text and images, as well as the original data with an interactive access point 1 1 1 See D for data and code availability. . 1. Introduction A knowledge graph (KG) represents real-world knowledge by sets of facts in the form of triples <h,r,t> , where the head entity (h) and tail entity (t) of a triple are two nodes of the graph, and the relation (r) is an edge connecting the two entities qualified by a predicate ( 23 ) . KGs are valuable background resources for various applications, e.g. recommendation ( 2 ) , web search ( 46 ) , question answering ( 57 ) , relation extraction ( 54 ) and RAG ( 26 ) . In the domain of cultural heritage, for example, paintings have various aspects such as name, creator, time, description, and image. The management of such extensive collections has become increasingly complex over the years and has posed significant challenges for curators and researchers, especially in AI-related applications. KGs have thus become a natural and promising approach to representing cultural heritage datasets ( 47 ; 13 ) , facilitating sophisticated querying and analytics, and providing a structured semantic context that is valuable for various applications. At the same time, international communities such as the GLAM Labs network and AI4LAM have emerged to support innovation and experimentation with digital cultural collections. GLAM Labs promotes collaborative environments where galleries, libraries, archives, and museums can explore new methods for working with digitized and born-digital heritage data ( 11 ) , while AI4LAM 2 2 2 https://sites.google.com/view/ai4lam/home (checked on 17/12/2025) provides a framework for sharing practices and developing AI-based approaches across the LAM sector. These initiatives reflect a broader movement toward leveraging multi-modal data and AI techniques to improve access, interoperability, and knowledge extraction in cultural heritage collections. Despite the usefulness of a structured representation, text and images, such as Wikipedia content, are rich and valuable resources for cultural heritage objects in an unstructured format ( 1 ; 41 ) . We believe that incorporating such information into the creation and extension of cultural heritage KGs can provide a deeper understanding of cultural connections and historical insights, thus facilitating research and application in this domain. Recognizing both the power of structured KGs and the diversity of unstructured sources, this paper introduces a new knowledge graph WJoconde, focusing on the French cultural heritage; it integrates multi-modal information including text descriptions and visual images. WJoconde covers 28k French cultural heritage entities, with more than 483k triples of 370 relationships. It is built from the Joconde database ( 35 ) - a French reference database on cultural heritage. WJoconde thus serves as a high-quality knowledge resource, the integration of texts and visual images further extends its usage and impact on various applications, such as querying and novel knowledge graph completion research. During the creation of WJoconde, we recognize one of the main problems with knowledge graphs, even large, well-established ones such as DBpedia ( 3 ) and Freebase ( 6 ) , which contain billions of facts, is that they are not complete. This is due to the fact that these KGs represent diverse and complex real-world knowledge, which is often constantly changing and sometimes ambiguous ( 29 ) . The same applies to existing knowledge graphs in the cultural heritage domain. The extensively studied Knowledge Graph Completion (KGC) methods are mostly embedding-based methods and are proven to achieve state-of-the-art link prediction performance ( 36 ; 45 ) . However, they follow the Closed World Assumption (CWA), which limits the entities to predefined sets. For cultural heritage KGs, where structured historical and cultural data are inherently incomplete and often hidden behind unstructured resources with high diversity, CWA will not be sufficient. Introducing new information to such KGs requires significant human expertise and effort. Thus, we argue that novel KGC methods under the Open World Assumption (OWA), where the prediction is not limited to known entities, are important to introduce new knowledge in KGs. We believe that methods involving multi-modal sources can help the extension of KGs under OWA in cultural heritage studies. To efficiently integrate these rich data sources to extend our KG, we propose a multi-modal KGC framework under OWA that uses the latest advancements of Large Language Models (LLMs) (eg, Llama 3 ( 22 ) , GPT ( 10 ) ) and Vision-Language Models (VLMs) (eg, BLIP ( 27 ) ). Our methodology applies LLMs for semantic annotation and contextual understanding of textual data. At the same time, VLMs are used to analyze and interpret images. Although the outputs of both models are uncontrolled and may correspond to preexisting KG entities, we propose a grounding pipeline to ensure the predicted information is indeed both novel and accurate. We evaluate our proposed method by querying and testing the degree of reliability of both models, along with human evaluation. Our experiments show that this framework can efficiently extract knowledge from different data sources in a zero-shot manner, enabling the extension of KGs under the OWA. Finally, we show that our proposed pipeline can increase the number of entities in the KG by 20%. Our approach is domain-independent, as it could be applied to any KG and unstructured external data sources, including text and images, for KGC under OWA. In this context, with the purpose of facilitating KGC research, we benchmark our dataset on various existing KGC models. The results are in Appendix A . Our contributions are summarized as follows: (1) We propose a novel cultural heritage KG: WJoconde, focusing on the French cultural heritage, with multi-modal data linked to heritage entities. WJoconde has more than 483k triples that aim to facilitate applications and research

Read the complete text