OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Onto logical U nderstanding, R easoning and L earning
Xiao Zhang, Huiyuan Lai, Qianru Meng, Johan Bos · 2025
Read OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Onto logical U nderstanding, R easoning and L earning by Xiao Zhang, Huiyuan Lai, Qianru Meng, Johan Bos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11031.txt.
Introduction
Xiao Zhang, Huiyuan Lai, Qianru Meng, and Johan Bos build OntoURL, a 2025 benchmark for language models on formal knowledge. Ontologies state a field's concepts and links in strict symbolic form. Models shine on loose text but are rarely tested on such strict structures.
The benchmark holds over 57,000 questions from 40 ontologies across eight fields. Its 15 tasks span grasping, reasoning, and learning from ontologies. Twenty open models were tested. They handled basic grasping fairly well but lagged on reasoning and learning. People beat models on most learning tasks. Prompt tricks shifted scores, which the authors use to map where models stand on symbolic work.
Catalog record
| CREATOR | Xiao Zhang, Huiyuan Lai, Qianru Meng, Johan Bos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 121 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11031
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OntoURL: A Benchmark for Evaluating Large Language Models on Symbolic Onto logical U nderstanding, R easoning and L earning Xiao Zhang, Huiyuan Lai, Qianru Meng, Johan Bos Abstract Large language models (LLMs) have demonstrated remarkable capabilities across a range of natural language processing tasks, yet their ability to process structured symbolic knowledge remains underexplored. To address this gap, we propose a taxonomy of LLMs’ ontological capabilities and introduce OntoURL , the first comprehensive benchmark designed to systematically evaluate LLMs’ proficiency in handling ontologies—formal, symbolic representations of domain knowledge through concepts, relationships, and instances. Based on the proposed taxonomy, OntoURL systematically assesses three dimensions: understanding, reasoning, and learning through 15 distinct tasks comprising 58,981 questions derived from 40 ontologies across 8 domains. Experiments with 20 open-source LLMs reveal significant performance differences across models, tasks, and domains, with current LLMs showing proficiency in understanding ontological knowledge but substantial weaknesses in reasoning and learning tasks. These findings highlight fundamental limitations in LLMs’ capability to process symbolic knowledge and establish OntoURL as a critical benchmark for advancing the integration of LLMs with formal knowledge representations. 1 Introduction Ontologies play a foundational role in encoding structured domain knowledge among the most prominent symbolic frameworks, particularly in fields such as finance, medicine, science and legal domain. They provide a formal structure through well-defined concepts (classes), relationships (e.g., hierarchies and semantic connections), and instances (individuals) gruber1993translation ; noy2001ontology ; mcguinness2004owl . In recent years, as Large language models (LLMs) have significantly revolutionized numerous fields with their remarkable capabilities in various tasks openai2023gpt4 ; wei-et-al-2022-chain , the interaction between ontologies and the knowledge-rich LLMs has sparked significant interest, raising research into ontology-related tasks such as leveraging LLMs for ontology creation. However, whether LLMs can truly comprehend and manipulate structured symbolic knowledge remains a subject of intense debate tang2023large ; pavlick2023symbols ; yan2024large ; saba2024reinterpreting . This discussion centers on whether statistical pattern recognition can replicate the symbolic representations and logical structures traditionally managed by classical knowledge representation systems. Therefore, a critical yet underexplored question arises for ontology practitioners and researchers: to what extent can LLMs understand, utilize, and construct ontologies? While several datasets have been developed for ontology-related tasks ( badie2017formal ; peng-etal-2022-copen ; wu-etal-2023-plms ; bombieri2024llms ; qin2024comprehensive ; song2025peeled ; he2023language ; sun2024large ; kang2024researcharena ; jiang2025hibench ; e2eol2024lo ; li2024ontology ) , these works typically focus on one or two isolated ontology aspects and are rarely designed specifically for evaluating LLMs. Furthermore, there is an absence of a comprehensive taxonomy that systematically categorizes the ontological capabilities required across domains and tasks, thus hindering the reliable evaluation of LLMs capabilities on ontology. To fill in this gap, we investigate three key research questions spanning three dimensions: (1) can LLMs accurately memorize the fine-grained details inherent in ontologies, including concepts, hierarchical relationships, properties, and instances? (2) Can LLMs perform robust reasoning over ontologies, such as transitive superclass inference or description logic reasoning? (3) Can LLMs autonomously construct ontologies based on their rich knowledge, such as tasks like ontology hierarchy construction? We first propose a taxonomy of ontological capabilities for LLMs and then introduce OntoURL , the first comprehensive benchmark for evaluating LLMs’ abilities on ontologies. OntoURL consists of 58,981 questions derived from 40 ontologies, encompassing 15 tasks across 8 domains. These tasks are meticulously crafted to assess LLMs’ proficiency in three crucial dimensions: understanding, reasoning, and learning. By evaluating 20 open-source LLMs, we gain critical insights into their strengths and limitations on handling structured symbolic knowledge. Our primary contributions are summarized as follows: • We present a taxonomy of LLM ontological capabilities in three dimensions: understanding, reasoning, and learning, providing a systematic framework for analyzing LLMs’ interactions with structured symbolic knowledge. • We introduce OntoURL , a benchmark comprising 58,981 questions derived from 40 ontologies, covering 15 tasks across 8 domains. Our benchmark enables rigorous evaluation on LLMs’ abilities across multiple dimensions, such as conceptual understanding, structural knowledge, logical reasoning, structure construction, and concept alignment. • We conduct in-depth pilot studies on 20 LLMs and provide comprehensive analyses across model scales, task levels, and specific domains. All code for benchmark construction and experiments, as well as the benchmark dataset, are publicly available (Appendix A ). 2 Background Ontology Ontologies provide formal, logic-based representations of domain knowledge, organizing concepts, properties, instances, and their interrelations in a structured, symbolic form gruber1993translation (Appendix C.1 ). Unlike taxonomies or controlled vocabularies which focus primarily on hierarchical groupings, or knowledge graphs which typically emphasize instance-level assertions, ontologies encode domain semantics through explicit axioms and logical constraints (further comparison in Appendix B.3 ). As a formal knowledge representation system, ontologies are typically formalized in Description Logic krötzsch2013descriptionlogicprimer —a family of formal languages designed for representing and reasoning about knowledge. Ontologies comprise three fundamental components: Terminological Box (TBox), containing class hierarchies and definitions; Assertional Box (ABox), capturing assertions about individuals; and Role Box (RBox), defining relationships and properties among entities. Figure 1 illustrates these components in a conference ontology excerpt, where “Chair” and its superclass “Committee Members” represent TBox elements, “Mary” as an instance of “Chair” forms part of the ABox, and “has_authors” is a relation between “Review” and “Review Expertise” belongs to the RBox. By offering a standardized vocabulary with formal semantics, ontologies support semantic interoperability, knowledge integration, and automated reasoning staab2013handbook . Most ontological resources are created by domain experts, such as Gene Ontology ashburner2000gene , Plant Ontology plant2002plant , and LKIF Core Legal Ontology hoekstra2007lkif . Appendix C.4 summarizes the 40 ontologies used in our benchmark, and Appendix C.3 introduces the language for describing these ontologies: RDF (a foundational data model) and OWL (the core ontology language). As ontologies have been central to symbolic AI approaches for decades, understanding and leveraging such structured symbolic knowledge are essential for LLMs. Figure 1 : A sub-ontology excerpt from the Conference Ontology for representing academic conferences, illustrating the hierarchical structure of classes (in green) and instances (in blue). Most of the classes, relations, instances, and semantic information are omitted for the clarity. Ontology-related Tasks Previous work has primarily focused on conceptual understanding, using probing techniques to examine how LLMs memorize and retrieve class-level knowledge badie2017formal ; peng-etal-2022-copen ; sahu2022unpacking ; patel2022mapping ; wu-etal-2023-plms ; shani-etal-2023-towards ; jang-lukasiewicz-2023-improving ; mitchell2023debate ; jin2024exploring ; song2025peeled , and structural knowledge he2023language ; mruthyunjaya2023rethinking ; park2024geometry ; jackermeier2024dual ; zhang-etal-2025-neural . Beyond basic understanding tasks, some works perform specialized forms of deductive logic reasoning on ontologies. Rule-based ontology reasoners support a wide range of inference tasks, including validating ontology coherence, deriving complete class hierarchies, assigning individuals to their most specific types, inferring property relationships, and executing queries to retrieve relevant classes or individuals tsarkov2006fact ; mendez2009reintroducing ; kazakov2012elk ; sertkaya2013elephant ; glimm2014hermit ; ceylan2015bayesian ; bobillo2016fuzzy ; fernandes2018graph ; balhoff2018arachne . Recently, a few studies have begun to explore using language models for ontology reasoning, particularly within the framework of Description Logic he2023language ; wang2024can . Another active research direction involves learning and constructing ontological structures. Traditional approaches employed statistical term extraction and pattern-based methods to identify candidate concepts and taxonomic relations hearst1998automated ; kietz2000method ; maedche2001ontology ; xu2002domain ; alfonseca2002unsupervised ; lonsdale2002peppering ; khan2002ontology ; biemann2005ontology ; asim2018survey ; xu2019automatic ; konys2019knowledge . More recent efforts leverage