Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

Tanay Aggarwal, Angelo Salatino, Francesco Osborne, Enrico Motta · 2024

Read Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field by Tanay Aggarwal, Angelo Salatino, Francesco Osborne, Enrico Motta (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08258.txt.

Introduction

Maps of research topics help scientists search, sort, and link a fast-growing literature, but drawing them by hand is slow and costly. Tanay Aggarwal, Angelo Salatino, Francesco Osborne, and Enrico Motta test in this 2024 paper whether large language models can do part of that work. The task is to judge how pairs of topics relate, a step at the heart of building such a map.

The team built a gold standard from an engineering word list and asked models to label topic pairs as broader, narrower, the same, or unrelated. They tested seventeen models, open and paid, large and small, under four styles of prompting with no examples. Scores were high for the best systems, and even small trimmed models did well once their settings were tuned. The results suggest machines can draft topic maps that humans then refine.

Catalog record

CREATORTanay Aggarwal, Angelo Salatino, Francesco Osborne, Enrico Motta
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08258

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field Tanay Aggarwal, Angelo Salatino, Francesco Osborne, Enrico Motta Abstract Ontologies of research topics are crucial for structuring scientific knowledge, enabling scientists to navigate vast amounts of research, and forming the backbone of intelligent systems such as search engines and recommendation systems. However, manual creation of these ontologies is expensive, slow, and often results in outdated and overly general representations. As a solution, researchers have been investigating ways to automate or semi-automate the process of generating these ontologies. One of the key challenges in this domain is accurately assessing the semantic relationships between pairs of research topics. This paper presents an analysis of the capabilities of large language models (LLMs) in identifying such relationships, with a specific focus on the field of engineering. To this end, we introduce a novel benchmark based on the IEEE Thesaurus for evaluating the task of identifying three types of semantic relations between pairs of topics: broader , narrower , and same-as . Our study evaluates the performance of seventeen LLMs, which differ in scale, accessibility (open vs. proprietary), and model type (full vs. quantised), while also assessing four zero-shot reasoning strategies. Several models with varying architectures and sizes have achieved excellent results on this task, including Mixtral-8×7B, Dolphin-Mistral-7B, and Claude 3 Sonnet, with F1-scores of 0.847, 0.920, and 0.967, respectively. Furthermore, our findings demonstrate that smaller, quantised models, when optimised through prompt engineering, can achieve strong performance while requiring very limited computational resources. Journal: Information Processing & Management 1 Introduction Ontologies of research topics, along with other knowledge organization systems (KOS), such as taxonomies and thesauri, play a vital role in organizing and retrieving information from digital libraries [ 1 ] . Major publishers and organizations in this domain, such as Elsevier, Springer Nature, ACM, IEEE, and PubMed, employ a variety of classification systems, including the All Science Journal Classification (ASJC), SpringerNature Taxonomy, ACM Computing Classification System 1 1 1 ACM Computing Classification System - https://dl.acm.org/ccs , the IEEE Theasurus, and Medical Subject Headings. These systems are crucial for enabling efficient information discovery and supporting effective scientific dissemination [ 2 , 3 , 4 ] . Furthermore, ontologies of research topics play a vital role in supporting a wide range of intelligent systems to navigate and interpret academic literature [ 5 , 6 , 7 ] . These include search engines [ 7 ] , conversational agents [ 8 ] , analytics dashboards [ 9 ] , and recommender systems [ 6 ] . Nevertheless, generating and maintaining ontologies of research topics is both expensive and time-consuming, typically requiring the involvement of several domain experts over an extended period of time [ 10 ] . Moreover, the ever-increasing volume of literature, with approximately 2.5 million new scientific papers published annually [ 11 , 12 ] , demands continuous updates to ensure that these ontologies accurately reflect the latest research developments. Over the past 25 years, Artificial Intelligence (AI) researchers have developed several methods for the automated or semi-automated generation of such ontologies, with varying degrees of success [ 13 , 14 , 10 , 15 ] . Some notable achievements include the semi-automatic pipeline for generating OpenAlex Topics [ 16 ] and the automatic development of the Computer Science Ontology [ 17 ] , along with a few other initiatives. However, most of the ontologies in this domain are still manually created because current solutions still struggle to generate large-scale, detailed representations of research topics. Consequently, this challenge remains far from being fully addressed. Over the past two years, Large Language Models (LLMs) have transformed the field of Natural Language Processing (NLP) by significantly improving machine comprehension of human language. In particular, some studies have demonstrated that LLMs can achieve remarkable performance even in zero-shot settings, where the model makes predictions without prior exposure to specific tasks or examples [ 18 ] . In this paper, we explore the potential of LLMs to facilitate ontology generation by identifying semantic relationships between research topics, with a specific focus on the field of engineering. This task is a fundamental step in the construction of scientific KOSs, and has traditionally been the most labour-intensive phase, often involving extensive deliberation among domain experts [ 10 ] . Specifically, we analyse the capability of several state-of-the-art LLMs to detect three semantic relations frequently adopted in such ontologies: broader , narrower , and same-as . In particular, our experimental analysis focuses on identifying such relations between research topics defined in the IEEE Thesaurus, the most widely adopted ontology in electrical and electronic engineering, which also offers broad coverage of computer science. To support this investigation, we introduce IEEE-Rel-1K, a gold-standard dataset derived from the IEEE Thesaurus, which describes the semantic relationships of 1,000 topic pairs. We evaluated 17 off-the-shelf LLMs on this dataset, varying in scale, accessibility (open or proprietary), and model type (full or quantised). The models included 3 full open models, 10 open quantised models, and 4 proprietary models. For each model, we employed four distinct zero-shot prompting strategies. The first strategy involved a straightforward prompt asking the model to determine the relationship between two topics, t_{a} and t_{b} . The second strategy extended this by applying the same method to both the original pair ( t_{a} , t_{b} ) and the reversed pair ( t_{b} , t_{a} ), with the results then being combined. The third strategy utilised a Chain-of-Thought (CoT) prompt, encouraging the model to reason through the relationship step by step. The fourth strategy was a two-way version of this CoT approach, where the process is repeated for both the original and reversed pairs before integrating the outcomes. Our findings show that several models exhibit strong zero-shot reasoning capabilities for this task. Proprietary models performed best, with Claude 3 Sonnet achieving an impressive F1 score of 0.967. Nonetheless, several open models also delivered highly competitive results, particularly when enhanced with advanced prompting techniques based on Chain-of-Thought (CoT) reasoning. Among the open models, the top performer was Dolphin-Mistral-7B (0.920 F1), a Mistral variant fine-tuned on the Dolphin dataset. The Mixtral model also demonstrated solid performance, reaching an F1 score of 0.847. Our study produced several notable insights. First, many models exhibit strong zero-shot reasoning capabilities for this task. Second, the performance of LLMs is highly dependent on the prompting strategy employed. In particular, the two-way variant of CoT prompting consistently outperforms alternative approaches. In fact, our results show that prompting strategies can lead to substantial performance gains, with improvements exceeding 0.2 F1 points in several instances. Finally, when the optimal prompting strategy described in this paper is adopted, smaller models, which are more scalable and cost-efficient, can achieve performance comparable to that of the best-performing architectures. These findings point to promising directions for the development of optimised and efficient models for ontology generation. Ultimately, our objective is to develop a detailed, comprehensive ontology of research topics across all domains. To achieve this, we plan to develop and publish field-specific ontologies using the SKOS standard 2 2 2 SKOS - https://www.w3.org/TR/2009/REC-skos-reference-20090818/ , subsequently mapping them together. This paper represents the first step in this ambitious direction. In summary, the contributions of this paper are

Read the complete text