Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation
Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li, Xi Yang · 2026
Read Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation by Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li, Xi Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06711.txt.
Introduction
Zhang and seven colleagues close the gap in reading oracle bone script. Treating each rare character as a closed-set image label fails, since characters combine a small set of recurring picture parts with meanings that carry across characters. Their agent-led setup grounds parts by sight, then chains part spotting, graph-based fact lookup, and link inference into sound readings. A new expert-checked set, OB-Radix, supplies 1,022 character images and 1,853 component images, and tests on three benchmarks show fuller, sharper readings than baselines.
Catalog record
| CREATOR | Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li, Xi Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41402 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.06711
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation Jianing Zhang, Runan Li, Honglin Pang, Ding Xia, Zhou Zhu, Qian Zhang, Chuntao Li, Xi Yang Abstract Deciphering ancient Chinese Oracle Bone Script (OBS) is a challenging task that offers insights into the beliefs, systems, and culture of the ancient era. Existing approaches treat decipherment as a closed-set image recognition problem, which fails to bridge the “interpretation gap”: while individual characters are often unique and rare, they are composed of a limited set of recurring, pictographic components that carry transferable semantic meanings. To leverage this structural logic, we propose an agent-driven Vision-Language Model (VLM) framework that integrates a VLM for precise visual grounding with an LLM-based agent to automate a reasoning chain of component identification, graph-based knowledge retrieval, and relationship inference for linguistically accurate interpretation. To support this, we also introduce OB-Radix, an expert-annotated dataset providing structural and semantic data absent from prior corpora, comprising 1,022 character images (934 unique characters) and 1,853 fine-grained component images across 478 distinct components with verified explanations. By evaluating our system across three benchmarks of different tasks, we demonstrate that our framework yields more detailed and precise decipherments compared to baseline methods. 1 Introduction Figure 1: Oracle Bone Script (OBS), a pictographic writing system of semantic components. Oracle Bone Script (OBS), the earliest known mature writing system in China, holds significant historical and cultural value. Of the more than 4,500 identified OBS characters, only approximately one-third have been deciphered, leaving a vast corpus of glyphs in mystery ( 19 ) . Each undeciphered character represents a lost fragment of ancient institutions, technologies, and beliefs. However, the fragmented and stylised nature of OBS inscriptions, coupled with the requirement for profound paleographic and contextual expertise, renders manual decipherment exceptionally difficult. In recent years, artificial intelligence has increasingly been leveraged for OBS interpretation ( 6 ; 38 ; 9 ; 15 ) . Most existing methods treat decipherment as a closed-set image recognition task, largely neglecting the structural, semantic, and contextual nuances intrinsic to the script. This narrow focus results in information waste and introduces interpretive biases, as models lack the domain-specific knowledge required to generalise beyond known characters. While Vision-Language Models (VLMs) have demonstrated robust general image–text understanding ( 24 ; 2 ) , their capacity for fine-grained perception and expert reasoning remains a bottleneck. In low-resource, specialised domains like OBS, standard VLMs often suffer from “visual hallucinations” or a lack of linguistic depth, leading to bad performance ( 4 ; 44 ) . Structurally, the OBS is an organised system of pictographic components, where each character is made up of discrete radicals that carry a distinct semantic weight and are frequently reused throughout the lexicon (Figure 1 ). This component-based architecture provides a critical logic bridge for decipherment: by identifying known pictographic components within an unknown glyph, we can systematically infer the meaning of characters that are new to the model. To leverage this, we propose an Agentic Retrieval-Augmented Generation (Agentic RAG) framework that empowers VLMs with component-based semantic augmentation (Figure 2 ). To support this, we introduce OB-Radix, a new expert-annotated dataset comprising 1,022 Oracle character images (934 unique characters) and 1,853 fine-grained component images (478 distinct components), each paired with expert-verified semantic explanations. Finally, to evaluate whether our approach achieves expert-level capability, we design three progressively advanced benchmarks: (1) component-level retrieval, (2) component relationship inference, and (3) OBS interpretation generation. Experimental results demonstrate that our framework outperforms baseline methods, providing a more interpretable and linguistically accurate pathway for OBS decipherment. In summary, our contributions are: • We reformulate oracle bone script (OBS) interpretation as a component-grounded, structure-aware reasoning task , rather than a purely visual recognition problem, and instantiate this formulation with a multimodal framework that integrates component-level visual cues and graph-based retrieval. • We construct OB-Radix, a component-level oracle bone script dataset, and build a knowledge graph that captures relationships among components, characters, and their semantic explanations, providing essential structured knowledge. • We design comprehensive evaluations to assess both the accuracy and interpretability of our approach. Results show that our framework produces interpretations closely aligned with expert annotations and that the multi-agent extension offers enhanced semantic grounding. Figure 2: Comparison of our proposed framework and baselines. We design an agentic RAG framework to integrate component-level knowledge for structured semantic augmentation of OBS. 2 Related Work Deciphering of Oracle Bone Script. Existing research relies on a single image morphology model to explore AI reading paths. 9 employs a diffusion approach to map oracle bone inscription images to modern Chinese characters, while 29 leverages image generation to provide visual interpretive guidance. However, the former lacks integration of textual semantics, and the latter results in incomplete understanding due to the absence of textual guidance. Other studies applied diverse AI techniques ( 6 ; 15 ; 38 ; 7 ) from different perspectives to aid in the decipherment of Oracle Bone Script. Graph Retrieval-Augmented Generation for VLMs. Although large-scale VLMs demonstrate strong zero-shot generalization, they still exhibit noticeable performance drops when the underlying training corpora lack or misrepresent the necessary domain knowledge ( 45 ; 25 ) . To enhance the specialization of visual language approaches in particular domains, Retrieval-Augmented Generation (RAG) approaches are employed ( 23 ; 46 ) . Unlike traditional fine-tuning, RAG dynamically retrieves relevant knowledge from external databases during inference, enabling VLMs to access domain-specific information on-demand without updating their pre-trained parameters. Additionally, to mitigate the potential noise present in general knowledge bases that may affect results, the concise representation provided by knowledge graphs are integrated, forming what is known as Graph RAG ( 28 ) . Oracle Bone Script Datasets. Most existing oracle bone script (OBS) datasets focus on character-level recognition, providing complete character images for end-to-end modeling, such as HUST-OBC ( 39 ) , EVOBC ( 8 ) , OBC306 ( 13 ) , Oracle-50k ( 11 ) , and HWOBC ( 18 ) . While these datasets cover multiple historical scripts, they lack component-level annotations and therefore provide limited support for structural decomposition and interpretable semantic analysis. OracleFusion ( 20 ) introduces radical-level structures, bounding boxes, and semantic concepts for oracle characters, but its annotations remain region-based and lack expert-curated component entities, consistent semantic interpretations, and explicit inter-component relations, limiting its support for component-grounded reasoning and knowledge graph construction. 3 OB-Radix Dataset Figure 3: Our annotation of an oracle character at the component level. We introduce OB-Radix, a dataset of hierarchical structural relations and grounded visual data meticulously curated by experts in paleography. Unlike prior character-level datasets ( 39 ; 8 ; 13 ; 11 ; 18 ) or those relying on ungrounded visual fragments ( 12 ) and text-only decompositions ( 14 ) , OB-Radix shifts the focus from the whole character to its constituent parts. This design enables models to learn and utilize the underlying compositional logic of the script, facilitating the interpretation of previously unseen or undeciphered glyphs through the identification of known pictographic elements. OB-Radix comprises 1,022 oracle character images covering 934 unique characters, along with 1,853 fine-grained component images spanning 478 distinct components. Three archaeology doctoral students were tasked with identifying components based on their paleographic function and meaning, rather than