Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering

Hanna Abi Akl · 2025

Read Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering by Hanna Abi Akl (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10249.txt.

Introduction

Hanna Abi Akl, writing alone, reports an early study on language models and formal knowledge in this 2025 paper. The work sits inside a doctoral project on ontology engineering, the craft of building structured vocabularies that machines can reason over. Language models are weak reasoners, and that weakness limits their use in such building work.

The opening experiments ask a narrow question: does the grammar used to state a logic problem change how well a small language model solves it? Early findings say a compact logical language can replace natural language without losing reasoning performance. The author hopes to build on that result, using small models to help draft ontologies that experts then refine.

Catalog record

CREATORHanna Abi Akl
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10249

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Investigating Language Model Capabilities to Represent and Process Formal Knowledge: A Preliminary Study to Assist Ontology Engineering Hanna Abi Akl Abstract Recent advances in Language Models (LMs) have failed to mask their shortcomings particularly in the domain of reasoning. This limitation impacts several tasks, most notably those involving ontology engineering. As part of a PhD research, we investigate the consequences of incorporating formal methods on the performance of Small Language Models (SLMs) on reasoning tasks. Specifically, we aim to orient our work toward using SLMs to bootstrap ontology construction and set up a series of preliminary experiments to determine the impact of expressing logical problems with different grammars on the performance of SLMs on a predefined reasoning task. Our findings show that it is possible to substitute Natural Language (NL) with a more compact logical language while maintaining a strong performance on reasoning tasks and hope to use these results to further refine the role of SLMs in ontology engineering. [orcid=0000-0001-9829-7401, [email protected], url=https://hannaabiakl.github.io/, ] keywords Small Language Models ,Logical Reasoning ,Knowledge Representation † † copyrightyear: 2025 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: RuleML+RR’25: Companion Proceedings of the 9th International Joint Conference on Rules and Reasoning, September 22–24, 2025, Istanbul, Turkiye † † address: Supervised by Fabien Gandon, Catherine Faron and Pierre Monnin Université Côte d’Azur, Inria, CNRS, I3S, Sophia Antipolis, France and Data ScienceTech Institute (DSTI), Paris, France 1 Introduction Since their introduction, Language Models (LMs) have achieved impressive results on a broad spectrum of Natural Language Processing (NLP) tasks, ranging from text classification to question-answering and text generation Guo et al. 2023 ; Minaee et al. 2024 . One field that still poses a challenge for LMs is reasoning. Over different kinds of reasoning tasks, LMs still struggle to achieve comparable results to humans in tasks involving explicit or implicit thinking (e.g. mathematical, commonsense, logical first-order reasoning) Guo et al. 2023 ; Liu et al. 2025 . These reasoning capabilities are essential for complex tasks such as knowledge graph construction or ontology engineering Wang et al. 2024 ; Jin et al. 2024 . For the latter, LMs have been used in various roles in the ontology learning pipeline, from requirement gathering to ontology creation, validation and alignment Lippolis et al. 2025 ; Zhao et al. 2024a ; Zhao et al. 2024b ; Zhao 2025 ; Fathallah et al. 2024 . Current research promotes neuro-symbolic approaches integrating symbolic mechanisms to curb LMs as current fine-tuning techniques (e.g. prompting) have proven insufficient in making them reliable reasoners Hou 2025 ; Schlegel et al. 2022 . In this perspective, the goal of our research in this PhD is to evaluate reasoning in LMs by focusing on small language models (SLMs) for the frugality, autonomy and privacy they support and by narrowing our evaluation to first-order logic (FOL) reasoning. We present and evaluate an experimental setup for testing the performance of SLMs on classifying the satisfiability of a logical conclusion given a set of premises. We also experiment with different knowledge representation languages to measure the impact of the formal representation of the input on the performance of SLMs on reasoning tasks. In doing so, we aim to address the following research question (RQ): RQ: Is there a better formal representation for logical data than natural language? This paper is structured as follows: Section 2 provides a review of the state of the art in LM reasoning methods in the context of formal problem resolution with a focus on logic tasks. Section 3 introduces our methodology and formalizes the Common Logic Grammar Construction (CLGC) pipeline. Section 4 presents our extended experimental setup and reports our findings with discussions on our results. Finally, Section 5 provides a summary of our key findings and outlines the next steps in this PhD research as well as potential future research directions. 2 State of the Art The focus of the first stage of this PhD is to combine formal knowledge representation with LM learning methods to improve LM performance on FOL reasoning tasks with the perspective of an application to ontology engineering. We report here the relevant research regarding state-of-the-art LM reasoning methods and logical reasoning applications. 2.1 Language Model Reasoning Reasoning is a fundamental aspect of human intelligence and is a cognitive process that involves the use of evidence, arguments and logic to arrive at conclusions Huang and Chang 2022 . The impressive performance of LMs on Natural Language (NL) data has led to observations that these models may exhibit reasoning abilities when they are sufficiently large Huang and Chang 2022 . However, reasoning also appears to be inherent in SLMs which are shown to be able to perform close to, and in some cases on par with, Large Language Models (LLMs). Approaches include extensive pre-training or guided techniques such as problem decomposition or the introduction of LLMs in SLM training processes for step correction as in the case of the SMART framework Srivastava et al. 2025 ; Bi et al. 2024 ; Kim et al. 2025 . In an effort to determine reasoning abilities in LMs, researchers have explored novelties in techniques, benchmark datasets and evaluation metrics Huang and Chang 2022 . Reasoning techniques largely fall under two pillars: LM-based and Reinforcement Learning (RL) techniques Besta et al. 2025 . LM-based reasoning techniques revolve around the methodologies spearheading model training, i.e. Supervised Fine-Tuning (SFT) on the one hand and Prompting and In-Context Learning on the other Huang and Chang 2022 ; Kumar et al. 2025 . Another emerging research direction is the exploration of hybrid or neuro-symbolic models which incorporate a form of symbolic knowledge into LMs to guide or enhance reasoning as seen in DomiKnowS and Logic Faghihi et al. 2021 ; Kesseli et al. 2025 . Different training techniques extending fine-tuning and prompting like COCONUT have resulted in LMs being used to tackle various reasoning tasks like mathematical, logical, causal, commonsense reasoning and coding Hao et al. 2022 . None of these methods however consider the potential impact of the logical formalism used to prompt the LMs. 2.2 Logical Reasoning Tasks for LMs Logical reasoning is an interesting domain of application for LMs since it subjects them to different types of reasoning, namely deductive, inductive, abductive or analogical Huang and Chang 2022 . Reasoning problems are classified in the literature into 2 main groups: Logical Question Answering (LQA) and Logical Consistency (LC) Cheng et al. 2025 . LQA applications encompass tasks that require LMs to generate the correct answer within complex logical problems which require sophisticated reasoning given a collection of premises and constraints Cheng et al. 2025 . LC applications require LMs to generate verifiable answers to complex questions without violating consistency (e.g. the model should not generate answers that contain contradictions) Cheng et al. 2025 . As we will show, our first research questions, methods and experimental setup fall under LQA and rely on FOLIO, a human-annotated benchmark dataset for FOL reasoning Han et al. 2024 . LQA models can be grouped into 3 categories depending on the approach. Solver-aided models rely on LMs translating NL problems into FOL before passing them

Read the complete text