RELRaE: LLM-Based Relationship Extraction, Labelling, Refinement, and Evaluation

George Hannah, Jacopo de Berardinis, Terry R. Payne, Valentina Tamma, Andrew Mitchell, Ellen Piercy, Ewan Johnson, Andrew Ng, Harry Rostron, Boris Konev · 2025

Read RELRaE: LLM-Based Relationship Extraction, Labelling, Refinement, and Evaluation by George Hannah, Jacopo de Berardinis, Terry R. Payne, Valentina Tamma, Andrew Mitchell, Ellen Piercy, Ewan Johnson, Andrew Ng, Harry Rostron, Boris Konev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03829.txt.

Introduction

George Hannah and a large team presented in 2025 a framework that uses large language models to label relationships hidden in laboratory data files. Robots in labs produce large volumes of XML data. Turning that data into shared knowledge graphs first requires naming the relationships implied by the file structure. Their system extracts candidate relationships, labels them, refines the labels, and then evaluates them.

The team investigated how accurately the models generate these labels in a lab automation setting. Their results show the models can support semi-automatic building of the schemas that underpin shared lab ontologies. The broader claim is that language models have a valuable role in ontology generation wherever expert time is scarce and data formats are messy.

Catalog record

CREATORGeorge Hannah, Jacopo de Berardinis, Terry R. Payne, Valentina Tamma, Andrew Mitchell, Ellen Piercy, Ewan Johnson, Andrew Ng, Harry Rostron, Boris Konev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03829

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RELRaE: LLM-Based Relationship Extraction, Labelling, Refinement, and Evaluation George Hannah, Jacopo de Berardinis, Terry R. Payne, Valentina Tamma, Andrew Mitchell, Ellen Piercy, Ewan Johnson, Andrew Ng, Harry Rostron, Boris Konev Abstract A large volume of XML data is produced in experiments carried out by robots in laboratories. In order to support the interoperability of data between labs, there is a motivation to translate the XML data into a knowledge graph. A key stage of this process is the enrichment of the XML schema to lay the foundation of an ontology schema. To achieve this, we present the RELRaE framework, a framework that employs large language models in different stages to extract and accurately label the relationships implicitly present in the XML schema. We investigate the capability of LLMs to accurately generate these labels and then evaluate them. Our work demonstrates that LLMs can be effectively used to support the generation of relationship labels in the context of lab automation, and that they can play a valuable role within semi-automatic ontology generation frameworks more generally. 1 Introduction Semi-structured data formats, including eXtensible Markup Language (XML) and Comma Separated Values (CSV), represent a vast amount of data on the web today [ 9 ] . These formats typically offer implicit semantics, connecting concepts within the documents. However, structured data formats like the Resource Description Framework (RDF) provide explicit, machine-readable semantics where relationships between concepts are clearly articulated. XML is usually defined by an XML Schema Document (XSD), whereas ontologies are used to define Knowledge Graphs (KGs). These ontologies (typically expressed in RDF Schema (RDFS) or the Web Ontology Language (OWL)) define concepts and their interrelations that allow inference and offer a richer view of the data within KGs [ 36 ] . Harnessing explicit semantics is crucial in data-intensive science. For example, analytical chemistry labs using robots generate vast quantities of data at an unprecedented rate over manual methods [ 37 ] . This data is often captured in XML-based formats like the Analytical Information Markup Language 1 1 1 AnIML core schema: https://www.animl.org/current-schema (AnIML) or the Allotrope Simple Model 2 2 2 https://www.allotrope.org/asm (ASM). While their XSDs (e.g., in AnIML) define structure (like Parent-Child or Element-Attribute links), the deeper meaning of scientific concepts frequently remains semantically underspecified. This semantic gap presents a challenge: elevating such implicitly structured scientific data to an explicit, knowledge-rich representation. Bridging this semantic gap involves translating XSDs into ontologies (e.g., OWL or RDFS). Existing methods [ 13 ] often use rule-based exploitation of XML’s hierarchical structure [ 5 , 6 , 14 , 41 ] . Whilst these approaches offer a starting point in the translation of XSD into OWL, the inherent KG-XML difference (explicitly structured vs. semi-structured) means effective XSD-to-ontology translation requires extensive knowledge and expertise in these domains. In traditional ontology engineering methods, this is done by a domain expert (DE) [ 27 , 31 ] , a process that can be time-consuming and prone to knowledge acquisition bottlenecks [ 12 , 26 ] . In this study, we aim to address the limitations of previous approaches by integrating Large Language Models (LLMs) into the XSD-to-ontology translation process, with the additional aim of reducing the workload of the DE and the ontology engineer. We present RELRaE (Relationship Extraction, Labelling, Refinement, and Evaluation), a novel framework for translating an XML schema into a foundational RDFS ontology. The basic ontology produced by RELRaE can be described as a skeleton ontology , representing the inter-concept relationships present in the XML schema input with injected domain knowledge. RELRaE operates in four distinct stages: (i) structural information representing hierarchical relationships between candidate concept pairs is extracted from the XML schema ( Concept Relationship Extraction ); (ii) labels for these relationships are then generated using a rule-based approach ( Rule-based Label Generation ); (iii) an LLM then refines these initial labels using schema-based contextual information ( Label Refinement ); and finally (iv) a different LLM, acting as a proxy for a domain expert, assesses the suitability of the refined label ( Automatic Label Evaluation ). This multi-stage process aims to produce a robust skeleton ontology, providing a strong starting point for further enrichment activities such as taxonomy completion [ 33 ] and ontology alignment [ 20 ] . To validate our approach, we conducted a comparative empirical evaluation using the AnIML schema, providing a complex benchmark from the analytical chemistry domain. This compared relationship labels generated by RELRaE’s hybrid approach against those from purely rule-based and LLM-only methods, using a gold-standard reference set created by domain experts. We also investigated the efficacy of an LLM in evaluating the generated labels by comparing its assessments against those of a domain expert. The findings indicate that RELRaE significantly enhances label accuracy compared to other methods and that LLMs demonstrate promise for the automated evaluation of these labels. The contributions of this paper are twofold: (i) RELRaE, a structured, multi-stage method that synergistically combines rule-based techniques with LLMs for the generation of skeleton ontologies from XML schemata; (ii) an empirical evaluation of RELRaE, focusing on its ability to generate accurate relationship labels and the viability of the LLM-as-a-judge approach for label assessment within a challenging, domain-intensive benchmark. Overall, this work demonstrates that the reuse of LLMs within a structured framework like RELRaE, can effectively support semi-automatic ontology generation, particularly in a complex domain. 2 Background and Related Work Several studies have explored the notion of enriching XSD to generate an OWL ontology, with many approaches being rule-based [ 13 ] . This relies on the assumption that the semantics present in the hierarchical structure of XML are implicit. Bohring and Auer [ 6 ] proposed one approach to translate XML schema directly into an OWL ontology via a mapping based pipeline, by identifying the structural patterns that define the elements and their relationships in the XML schema, and mapping them to a corresponding OWL pattern. Each relationship identified in the source schema is classed as either a partOf or a subClassOf relation, and labels for these relationships are formed by attaching either the “ has ” or “ dtp ” prefix to the textual label of the entity. Whilst their rules can perform well in situations where the source XSD is well formed and where standard structural patterns are used to represent the relationships between concepts, this is not always the case, and the relationships identified do not necessarily correlate with the intentions of the schema author. These limitations can be addressed through the injection of additional knowledge, thereby enriching the knowledge inherent in the XSD to construct a new ontology. Within traditional ontology engineering methodologies this knowledge is typically provided by the domain expert [ 27 ] ; however, this process can be significantly time consuming and suffers from the problem of the knowledge acquisition bottleneck. The rise in popularity of LLMs since the release of ChatGPT-3 in late 2022 3 3 3 https://openai.com/index/chatgpt/ has resulted in a number of investigations into the viability of utilising LLMs as part of the ontology engineering process [ 1 , 2 , 17 , 18 , 22 , 28 , 30 ] . Due to the large volume of training data used in their creation, LLMs

Read the complete text