Human-In-The-Loop Workflow for Neuro- Symbolic Scholarly Knowledge Organization
Lena John, Ahmed Malek Ghanmi, Tim Wittenborg, Sören Auer, Oliver Karras · 2025
Read Human-In-The-Loop Workflow for Neuro- Symbolic Scholarly Knowledge Organization by Lena John, Ahmed Malek Ghanmi, Tim Wittenborg, Sören Auer, Oliver Karras (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03221.txt.
Introduction
Lena John, Ahmed Ghanmi, Tim Wittenborg, Sören Auer, and Oliver Karras present ExtracTable in work from 2025. Growing publication volume makes it hard to turn papers into structured knowledge by hand. Their human in the loop workflow helps researchers reshape plain publications into tables that fit user defined schemas.
Large language models handle steps such as preprocessing and data extraction, while users validate the output. Results feed into knowledge graphs, with tests run in the Open Research Knowledge Graph community. Users rated usability very highly on a standard scale. Time from a research interest to literature based insights fell sharply in the trial. Human checks remain central for quality and for links to existing resources.
Catalog record
| CREATOR | Lena John, Ahmed Malek Ghanmi, Tim Wittenborg, Sören Auer, Oliver Karras |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-03221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Human-In-The-Loop Workflow for Neuro- Symbolic Scholarly Knowledge Organization Lena John, Ahmed Malek Ghanmi, Tim Wittenborg, Sören Auer, Oliver Karras Abstract As the volume of scientific literature continues to grow, efficient knowledge organization is an increasingly challenging task. Traditional structuring of scientific content is time-consuming and requires significant domain expertise, increasing the need for tool support. Our goal is to create a Human-in-the-Loop (HITL) workflow that supports researchers in creating and structuring scientific knowledge, leveraging neural models and knowledge graphs, exemplified using the Open Research Knowledge Graph (ORKG). The workflow aims to automate key steps, including data extraction and knowledge structuring, while keeping user oversight through human validation. We developed a modular framework implementing the workflow and evaluated it along the Quality Improvement Paradigm (QIP) with participants from the ORKG user community. The evaluation indicated that the framework is highly usable and provides practical support. It significantly reduces the time and effort required to transition from a research interest to literature-based answers by streamlining the import of information into a knowledge graph. Participants evaluated the framework with an average System Usability Scale (SUS) score of 84.17, an A+ – the highest achievable rating. They also reported that it improved their time spent, previously between 4 hours and two weeks, down to an average of 24:40 minutes. The tool streamlines the creation of scientific corpora and extraction of structured knowledge for KG integration by leveraging LLMs and user-defined models, significantly accelerating the review process. However, human validation remains essential throughout the extraction process, and future work is needed to improve extraction accuracy and entity linking to existing knowledge resources. 1 Introduction Conducting literature reviews is indispensable for scientific research. This task is inherently challenging, as it requires a significant amount of time and effort to gain an overview of the existing research and knowledge on a specific topic, helping to support methodology, contextualize research, or identify gaps in knowledge [ 33 ] . With the rapid growth in scientific discovery and distribution of knowledge, it has become even more difficult to produce high-quality reviews [ 23 ] . Recent advancements in artificial intelligence (AI), particularly the rise of large language models (LLMs) and AI-based Literature Reviews (AILR) [ 39 ] , offer promising opportunities to streamline and automate repeatable and time-consuming processes [ 29 ] . However, despite their growing popularity, AI models face significant limitations, primarily the tendency to produce hallucinations, resulting in inconsistencies and confusion [ 28 ] . Combining AI capabilities with a consistent knowledge base, called neuro-symbolic AI, mitigates these risks by leveraging reliable knowledge representations to alleviate AI’s unreliability [ 3 ] . The knowledge bases need to be Findable, Accessible, Interoperable, and Reusable (FAIR) [ 40 ] to address the requirements of the scholarly domain and AI technology [ 2 ] . Such solutions already exist, like the Open Research Knowledge Graph (ORKG) 1 1 1 https://orkg.org/ and its neuro-symbolic service Ask [ 3 ] , offering a novel approach by representing scientific information in a structured format, providing both human- and machine-readability. However, organizing knowledge at scale reintroduces the original challenge of requiring human oversight, leading to manual work and a circular dependency. To address this issue, an iterative, Human-in-the-Loop (HITL), AI-assisted knowledge organization approach is needed, constantly improving scholarly knowledge representation. Our contributions include 1) A HITL workflow, combining LLM support with Natural Language Processing (NLP) techniques to assist users by streamlining, automating, and formalizing tasks of literature reviews. 2) A modular framework implementing this workflow to build a scientific corpus of literature and transform information into structured knowledge, ready for integration into knowledge graphs. 3) An iterative evaluation with nine researchers over 4 months, developing and validating the frameworks’ capacity to ensure greater efficiency while allowing users to maintain oversight of the automated processes. This work is structured as follows: In section 2 we give an overview of the background and related work in the HITL context. Our approach is described in section 3 , followed by an outline of the implementation in section 4 . We evaluate the developed solution in section 5 and discuss the results in section 6 . We conclude our work in section 7 . 2 Background and Related Work This section explores Knowledge Graphs (KGs), AI-driven automation, and HITL systems in scholarly knowledge organization, highlighting their roles in literature reviews and knowledge integration. Knowledge Organization. Knowledge Graphs (KGs) are designed to represent and interconnect information in a structured, machine-readable format [ 36 ] , like the scientific knowledge graph (SKG) Semantic Scholar [ 38 ] . Manually preparing data for KGs, particularly SKGs, is time-consuming and lacks scalability, as it requires domain specialists to search, interpret, and structure scientific literature [ 12 ] . Without domain experts, there is a higher risk of integrating incorrect, incomplete, or outdated information into the SKG, reducing its reliability and quality [ 15 ] . The quality of SKGs heavily depends on their construction method, with significantly different results between manual and automated creation methods [ 13 ] . The ORKG is a versatile scientific knowledge platform that uses crowdsourcing to organize and link scientific knowledge across various topics and domains [ 3 ] . Karras et al. [ 18 , 19 , 20 ] demonstrated how the ORKG can enhance and ensure FAIR [ 40 ] literature reviews, highlighting their importance in providing research with structured, long-term available data. The ORKG Assistant for Scientific Knowledge (Ask) [ 3 ] applies a neuro-symbolic approach using semantic search, LLMs, and KGs. It exemplifies a next-generation scholarly search and exploration system, combining semantic knowledge representation with AI. Artificial Intelligence (AI). AI and NLP techniques can automate content generation and assignment [ 10 ] , but these methods often lack transparency and accountability while struggling with semantic context [ 21 ] . Fully relying on AI-supported technologies poses significant risks, making human supervision and validation indispensable [ 30 ] . The study conducted by Danler et al. [ 9 ] demonstrates that, while AI-generated outputs can be of high quality, “ relying solely on AI-generated outputs for crafting a scientific paper is not advisable ” [ 9 , p. 207] . Bacinger et al. [ 5 ] present Litre Assistant, a machine-learning-based solution to determining a corpus of scientific papers as the initial task of the Systematic Literature Review (SLR) process. Their work introduces a system that allows users to iteratively select desired papers as a set of training data for machine learning (ML) algorithms for an automated classification process. Ye et al. [ 43 ] propose a hybrid, “human-oriented”, but AI-based workflow to support specific steps of an SLR. Their workflow screens full-text articles to extract relevant data and allows the user to perform inclusion and exclusion decisions, resulting in a spreadsheet of LLM-generated data. The work of Alshami et al. [ 1 ] introduces a method where humans review ChatGPT-generated data for literature setup, abstract screening, data extraction, and analysis. However, relying solely on ChatGPT restricts access to real-time data, limiting the research scope to the model’s training data and reducing its applicability. Human-in-the-Loop (HITL). Several previously mentioned works incorporate human oversight in selection, classification, processing, and representation to achieve more accurate