Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin,

Xiao Zhang, Johan Bos · 2025

Read Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, by Xiao Zhang, Johan Bos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04377.txt.

Introduction

Xiao Zhang and Johan Bos teach vision models to read tombstones into structured meaning. Their 2025 paper deals with grave markers, dense with life facts, craft, and communal memory, yet worn by erosion, harm, and weather. Their frame turns a tombstone photo into a Tombstone Meaning Representation, holding both seen carving and read text. Retrieval adds outside facts, such as place names and job codes, to fill gaps.

Against a classic OCR chain, parsing scores leapt from low to high. Checks spanned tongues and cultures, and fake wear tested grit on damaged stones. The authors frame the work as a first formal step for heritage care.

Catalog record

CREATORXiao Zhang, Johan Bos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04377

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions Conference: Proceedings of the 33rd ACM International Conference on Multimedia; October 27–31, 2025; Dublin, Ireland Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland DOI: 10.1145/3746027.3755444 ISBN: 979-8-4007-2035-2/2025/10 CCS: Computing methodologies Natural language generation CCS: Computing methodologies Image representations Xiao Zhang, Johan Bos Abstract . Tombstones are historically and culturally rich artifacts, encapsulating individual lives, community memory, historical narratives and artistic expression. Yet, many tombstones today face significant preservation challenges, including physical erosion, vandalism, environmental degradation, and political shifts. In this paper, we introduce a novel multi-modal framework for tombstones digitization, aiming to improve the interpretation, organization and retrieval of tombstone content. Our approach leverages vision-language models (VLMs) to translate tombstone images into structured Tombstone Meaning Representations (TMRs), capturing both image and text information. To further enrich semantic parsing, we incorporate retrieval-augmented generation (RAG) for integrate externally dependent elements such as toponyms, occupation codes, and ontological concepts. Compared to traditional OCR-based pipelines, our method improves parsing accuracy from an F1 score of 36.1 to 89.5. Furthermore, we evaluate the model’s robustness across diverse linguistic and cultural inscriptions, and simulate physical degradation through image fusion to assess performance under noisy or damaged conditions. Our work represents the first attempt to formalize tombstone understanding using large vision-language models, presenting implications for heritage preservation. The code and supplementary materials are available at: https://github.com/LastDance500/Tombstone-Parsing . Figure 1. The overall idea of tombstone parsing. The input is an image of a tombstone, and the output is semantically interpreted, structured information, that can be stored in a relational database or semantic graph. † † cc-license: by 1. Introduction Tombstone inscriptions give us rich historical and cultural insight. Therefore, cemeteries have sparked the interest of many different types of researchers, including anthropologists, archaeologists, genealogists, historians, philologists, sociologists, and theologists ( Saller and Shaw 1984 ; Eckert 1993 ; Carmack 2002 ; Veit and Nonestied 2008 ; Long 2016 ; Bos 2022 ; Chen et al. 2024 ) . Besides the names of the deceased and dates and location of birth and death, tombstones often also contain information about family relationships, occupations, epitaphs, biblical quotes, poetry, and more. However, the intricate nature of these tombstones makes it challenging to interpret, organize and retrieve the valuable information they contain. Moreover, many tombstones are at risk due to physical erosion, vandalism, environmental degradation, and political shifts ( Matthias 1967 ; Streiter et al. 2007a ) . Digital preservation is therefore essential—not only to safeguard these irreplaceable cultural records but also to facilitate storage and search in structured databases. With the ultimate aim of creating a large database with structured information of tombstone inscriptions, we investigate the possibility of automatically parsing tombstone images into structured data suitable for storage, in order to assist data curators and provide cemetery researchers with a powerful tool for searching into databases of digitized tombstones. The overall idea can be seen in Figure 1 , where the input is an image (photograph) of a tombstone, and the output is an entry in a relational database. 1 1 1 Names of people on tombstones shown in this paper have been masked. Our aim is to stimulate research in developing automatic tools for cemetery research rather than distributing private information online. Cemetery researchers are then able to systematically search tombstones on dates, toponyms, gender, jobs, relationships, and other features. Our work has points of contact with the XML-based annotation scheme for graveyards and tombstones proposed by Streiter et al. 2007b . However, their scheme focuses on the physical properties of tombstones, whereas we aim to analyse the content of tombstone inscriptions. This content includes people, dates, locations, symbols, religious references that are mentioned on tombstones and the relations between them (temporal relations, family relations, occupational relations). The research by Franken and van Gemert 2013 , who developed a system for automatically recognising and interpreting ancient Egyptian hieroglyphs from photographs based on a dataset of about 4,000 hieroglyphs, is similar in objectives to our research, and so is other research in this area ( Zhang 2024 ; Assael et al. 2022 ; Jiang et al. 2024 ) . But even closer to our goal is earlier work by Bos et al. 2022 , who propose a pipeline-based approach consisting of image segmentation, OCR-based text extraction, and structured parsing. Their approach requires extensive manual annotation of relevant image segments, and has proven to be limited in effectiveness, achieving F1 scores below 40%, even for a simplified version of the task. As a result, it is not scalable to real-world scenarios. Thus, in this paper, we investigate whether we can take advantage of recent advances in end-to-end vision-language models for this task. Vision-Language Models (VLMs) ( OpenAI 2024 ; Anthropic 2024 ; Gemini 2024 ; Qwen 2025 ) have demonstrated remarkable capabilities in image and text comprehension, enabling direct generation of natural language descriptions from images. This presents an opportunity to improve tombstone parsing by directly generating structured content from images of gravestones rather than relying on OCR-based methods in complex and vulnerable pipeline architectures. However, VLMs lack domain-specific knowledge—they do not inherently interpret human-defined identifiers, which are essentially arbitrary labels lack of intrinsic meaning, such as geocodes (location identifiers for toponyms), HISCO codes (occupational classification) and WordNet sense number (grounded meaning of a word). Drawing inspiration from prior work on retrieval-augmented generation (RAG) for formal textual semantic parsing ( Zhang et al. 2024a ) , we explore the way to incorporate RAG into the VLM pipeline to enable the retrieval and integration of relevant external knowledge. To this end, we propose the Tomb2Meaning (T2M) framework, which combines VLMs and RAG to convert tombstone images into formal, structured meaning representations. Our investigation is guided by the following three research questions in the context of automated tombstone parsing: • Can VLMs be leveraged to enhance tombstone parsing, outperforming previous OCR-based methods? • Can RAG bridge the knowledge gap in neural tombstone parsing? • How do models handle challenging stones, such as those with linguistically complex inscriptions or physical defects? Following a review of background and related work on semantic parsing and tombstone meaning representations (Section 2 ), we introduce the T2M framework (Section 3 ). We then present the experiments (Section 4 ) that evaluate VLMs and RAG under few-shot and fine-tuned settings, including three strategies for integrating retrieval components. Our results (Section 5 ) show that fine-tuned VLMs achieve the state-of-the-art performance in tombstone parsing, with RAG further boosting accuracy by incorporating external domain knowledge. In Section 6 , we assess the models on linguistically challenging inscriptions across five dimensions, and evaluate robustness under simulated physical degradation. 2. Background and Related Work 2.1. Automated Formal Semantic Parsing Formal meaning representations play a crucial role in advancing natural language understanding (NLP) by providing structured, abstract depictions of sentence meaning. Abstract Meaning Representation (AMR) ( Banarescu et al. 2013 ) utilizes directed acyclic graphs to encapsulate the core semantic relationships within a sentence. Discourse Representation Structure (DRS) ( Kamp and Reyle 1993 ; Bos et al. 2017 ; Bos 2023 ) is designed to capture more context-dependent phenomena such as anaphora, temporal expressions and quantification across extended

Read the complete text