Leveraging Large Language Models for Automated Definition Extraction with TaxoMatic - a Case Study on Media Bias

Timo Spinde, Luyang Lin, Smi Hinterreiter, Isao Echizen · 2025

Read Leveraging Large Language Models for Automated Definition Extraction with TaxoMatic - a Case Study on Media Bias by Timo Spinde, Luyang Lin, Smi Hinterreiter, Isao Echizen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00343.txt.

Introduction

Spinde, Lin, Hinterreiter, and Echizen introduce TaxoMatic, a framework for pulling definitions from research papers with language models, in a 2025 paper. Definitions are the base of taxonomies, yet gathering them by hand is slow. Media bias serves as the test domain.

The pipeline collects papers, sorts them for relevance with a language model, then extracts clear definitions of concepts. It was tested on a data set of thousands of hand-rated articles. One leading model gave the best results on both sorting and extraction steps. The authors plan wider data sets and new domains. The work shows language models can speed taxonomy building.

Catalog record

CREATORTimo Spinde, Luyang Lin, Smi Hinterreiter, Isao Echizen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging Large Language Models for Automated Definition Extraction with TaxoMatic - a Case Study on Media Bias Timo Spinde, Luyang Lin, Smi Hinterreiter, Isao Echizen Abstract Defining complex, evolving concepts in academic research and extracting clear taxonomies from many publications is challenging. To streamline systematic reviews and capture shifts in conceptual understanding, we present our ongoing work on TaxoMatic - a framework leveraging Large Language Models (LLMs) to automate definition extraction from academic literature. The framework encompasses data collection, relevance classification to identify papers with definitions, and definition extraction using LLMs. As a first case study, we tested our relevancy evaluation component on 2,398 articles on media bias, a domain particularly rich in varying definitions and sub-concepts. Then, we evaluated our definition extraction component on manually reviewed papers, yielding 123 definitions from 113 relevant articles. Among five tested LLMs, Claude-3-sonnet achieved the highest F1 score (0.381) for relevance classification and demonstrated a median cosine similarity of 0.557 for definition extraction with role prompting. Future directions include improving relevance classification, expanding ground truth datasets, and applying this framework to other domains, potentially enhancing conceptual clarity across disciplines. 1 National Institute of Informatics, Japan 2 The Chinese University of Hong Kong, China 3 University of Würzburg, Germany [email protected], [email protected], [email protected], [email protected] Code/Dataset — https://github.com/Media-Bias-Group/Taxomatic 1 Introduction Defining concepts and building taxonomies is a foundational research task, as it ensures methodological clarity and facilitates interdisciplinary communication ( Spinde et al. 2023 ) . However, extracting clear, systematic definitions from academic literature remains a significant challenge across domains, especially given the growing complexity of concepts and the rapid expansion of research output ( Fel et al. 2024 ) . Recent advances in large language models (LLMs) present new opportunities to streamline definition identification in academic research ( Banerjee, Chakravarthi, and McCrae 2024 ) . Although LLMs have been explored for various information extraction and analysis tasks, their application to systematically extracting definitions and supporting conceptual clarity has not yet been largely investigated ( Banerjee, Chakravarthi, and McCrae 2024 ) . This study proposes and evaluates initial steps toward building TaxoMatic, a framework for automated definition extraction using LLMs. To assess its feasibility and reliability, we apply the individual parts of TaxoMatic to the domain of media bias, a concept widely studied in communication, political science, and computational linguistics but still lacking universally accepted definitions ( Spinde et al. 2022 ; Wessel et al. 2023 ; Horych et al. 2024 ) . We address the following research questions: • RQ1 : How accurately can LLMs evaluate the relevance of academic publications on media bias compared to human assessments? • RQ2 : How do LLM and human-extracted definitions compare in content and semantic similarity? We developed a three-stage workflow encompassing relevance classification, definition extraction, and evaluation. To create a ground truth, we collected 75,151 related scientific publications and manually rated 2,398 for relevancy to media bias research. From 113 deemed relevant, we manually extracted 123 definitions ( Section 3 ). The key contributions of this work include: • Presenting an LLM-based process for systematic definition extraction from academic literature. • Demonstrating the process’s utility in extracting definitions of complex phenomena, focusing on media bias. • Providing a dataset to evaluate future frameworks focusing on relevancy analysis and definition extraction. To ensure the dataset adheres to FAIR principles, we make all resources used in the process available (see the link after the abstract), use persistent identifiers for future updates, and maintain open, standardized formats to enhance interoperability and reusability. 2 Related Work Definitions are the core of academic research, providing consistent communication and interpretation. When well established, researchers can engage in shared dialogue and consistently explore the same phenomenon ( Navigli, Velardi, and Ruiz-Martínez 2010 ) . Many domains, especially those studying human behavior, culture, or communication, face challenges in agreeing on unified definitions ( Spinde et al. 2023 ) . The phenomena they examine are often subjective, context-dependent, and influenced by multiple factors. For instance, in sociology and political science, concepts like democracy or social justice vary across cultural and ideological contexts. Similarly, media studies lack consensus on media bias ( Spinde 2021 ) , with some focusing on visible bias like partisan reporting ( Milyo and Groseclose 2005 ) , while others define media bias as a linguistic concept ( Spinde et al. 2021a ) . The definitional fragmentation complicates comparison across studies ( Spinde et al. 2023 ) . Despite their importance, clear definitions are difficult to establish. Systematic reviews, which gather, analyze, and synthesize literature, often begin taxonomy-building but are labor-intensive and subjective. Researchers’ biases can influence the definitions, creating inconsistencies across studies ( Krippendorff 2019 ) . Additionally, manual analysis becomes impractical as datasets grow, slowing taxonomy development. Traditional computational approaches for term extraction struggle with open-text documents due to their inability to effectively handle unstructured formats and context-dependent relationships ( Bovi, Telesca, and Navigli 2015 ) . Advances in LLMs like GPT ( Brown et al. 2020 ) and Mistral ( Jiang et al. 2023 ) , built on transformer architectures, now enable automated definition extraction and support tasks like taxonomy building. LLMs excel at in-context learning, outperforming fine-tuned and unsupervised extractors in various domains, and their performance improves significantly with careful prompt engineering and experimenting with various prompt strategies ( Banerjee, Chakravarthi, and McCrae 2024 ) . Still, their exact quality and reliability in extracting definitions and building taxonomies are yet unclear ( Banerjee, Chakravarthi, and McCrae 2024 ) . Challenges persist, especially in domains with ambiguous or evolving terminologies and issues like hallucinations or dependency on predefined taxonomies ( Banerjee, Chakravarthi, and McCrae 2024 ) . Various datasets exist to explore definition extraction. The WCL dataset includes 5,000 sentences of explicit definitions from Wikipedia, limited to structured content ( Navigli, Velardi, and Ruiz-Martínez 2010 ) . The DEFT corpus, for SemEval-2010, covers 4,000 annotated sentences but lacks implicit or evolving definitions ( Spala et al. 2020 ) . DefIE contains 85,000 definitions from free-text sources across diverse domains ( Bovi, Telesca, and Navigli 2015 ) . General dictionaries, like Oxford or Urban Dictionary 1 1 1 Which also exhibit inconsistent quality and informal style. , collect various kinds of definitions ( Oxford University Press 2025 ; Urban Dictionary 2025 ) . Despite their value, all these datasets have limitations: (1) a lack of implicit or contested definitions (DEFT) ( Spala et al. 2020 ) , (2) focus on pre-structured content (WCL) ( Navigli, Velardi, and Ruiz-Martínez 2010 ) , (3) inconsistent quality (Urban Dictionary) ( Urban Dictionary 2025 ) , (4) limited adaptability to complex domains (all), and (5) no focus on academic definitions (all). These issues hinder their use for detailed, context-rich tasks, prompting the creation of our new dataset. 3 Methodology We show the general workflow of our TaxoMatic framework and the underlying case study in Table 1 . Our process includes multiple steps of a systematic literature search: (1) A keyword-based search for literature, (2) an assessment of which of the search results are relevant to the topic at hand, and (3) the extraction of required information — in our case, definitions. Two intermediate steps are required to process this data with LLMs and to create

Read the complete text