ExOAR: Expert-Guided Object and Activity Recognition from Textual Data

Iris Beerepoot, Vinicius Stein Dani, Xixi Lu · 2025

Read ExOAR: Expert-Guided Object and Activity Recognition from Textual Data by Iris Beerepoot, Vinicius Stein Dani, Xixi Lu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03790.txt.

Introduction

Process mining studies how work actually flows inside an organisation, but it needs structured logs, while much evidence sits in free text. Iris Beerepoot, Vinicius Stein Dani, and Xixi Lu present ExOAR in a 2025 preprint on databases and society. Their method targets object-centric process mining, where records must name the objects involved and the activities performed on them.

ExOAR pairs a large language model with a human checker in a staged workflow. The model proposes object types, activities, and individual instances, guided by context such as the user's profession. The user reviews and corrects each round before the next begins. After a demonstration, the tool was tested on real window-tracking data from five users. It turned messy text into structured logs with clear meaning, while keeping flexibility and human oversight throughout.

Catalog record

CREATORIris Beerepoot, Vinicius Stein Dani, Xixi Lu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03790

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ExOAR: Expert-Guided Object and Activity Recognition from Textual Data Thanks: Accepted manuscript on August 22, 2025, to the 2nd International Workshop on Generative AI for Process Mining (GenAI4PM 2025), held in conjunction with the 7th International Conference on Process Mining (ICPM 2025). Iris Beerepoot, Vinicius Stein Dani, Xixi Lu Abstract Object-centric process mining requires structured data, but extracting it from unstructured text remains a challenge. We introduce ExOAR (Expert-Guided Object and Activity Recognition), an interactive method that combines large language models (LLMs) with human verification to identify objects and activities from textual data. ExOAR guides users through consecutive stages in which an LLM generates candidate object types, activities, and object instances based on contextual input, such as a user’s profession, and textual data. Users review and refine these suggestions before proceeding to the next stage. Implemented as a practical tool, ExOAR is initially validated through a demonstration and then evaluated with real-world Active Window Tracking data from five users. Our results show that ExOAR can effectively bridge the gap between unstructured textual data and the structured log with clear semantics needed for object-centric process analysis, while it maintains flexibility and human oversight. 1 Introduction The availability of unstructured textual data from user interactions, such as window titles, document names, and application metadata, offers new opportunities for process analysis 3 ; 11 . However, converting this data into structured formats suitable for object-centric process mining remains a significant challenge. A key challenge lies in identifying the objects and activities embedded in the text. This challenge is evident across a variety of real-world domains. In healthcare, clinical case management depends on free-text sources like physician notes and discharge summaries, where extracting meaningful information about patients, treatments, and procedures requires expert knowledge due to domain-specific language and context 12 . In IT service management, helpdesk systems generate large volumes of tickets and chat logs containing informal and evolving terminology, making it difficult to consistently identify issue types and resolution steps 8 . In academic settings, digital traces like window tracking logs or file names encode complex, personal workflows that only the individual involved can meaningfully interpret 3 . Similarly, in industrial manufacturing, maintenance logs and shift reports rely on technical shorthand and equipment codes known only to local staff 14 . These examples underscore a shared limitation: fully automated extraction methods often fall short due to ambiguity or domain specificity, while fully manual approaches are too labor-intensive to scale. Recent progress in large language models (LLMs) has enabled new methods for semantic interpretation 13 ; 24 ; 25 . While LLMs can suggest meaningful labels and associations, their outputs are often inconsistent or insensitive to domain-specific nuances 20 . At the same time, fully manual labeling remains time-consuming and error-prone 21 . We introduce ExOAR (Expert-Guided Object and Activity Recognition), an interactive method that integrates the use of an LLM with expert validation. Users provide contextual input and the LLM is prompted to generate suggestion for candidate object types, activities, and objects. These suggestions are then reviewed and refined by the user and then used to enrich events, enabling object-centric process mining. ExOAR’s novelty lies in its human-in-the-loop design, making it especially effective where automated extraction alone falls short. We implement ExOAR as a practical tool and demonstrate its use on textual data from one of the authors. While the case study focuses on a specific data source, the approach is data-agnostic and extensible. A preliminary evaluation with another two academics and two professionals from outside of academia further illustrates its usability and effectiveness. 2 Background and Related Work This section outlines key concepts in object-centric process mining and the challenges of extracting structured event data from unstructured sources. We also review recent efforts using LLMs for semantic recognition. 2.1 Object-Centric Process Mining Object-centric process mining (OCPM) has emerged as a set of techniques that analyze processes in complex, multi-entity scenarios. Unlike traditional process mining techniques, OCPM allows for the mining of processes involving multiple object types and their interactions. Each event can be associated with one or more objects of different types, capturing the rich relational structure in many real-world processes 23 . A prerequisite for object-centric process mining is the availability of a structured, object-centric event log (OCEL), in which the events are explicitly linked to relevant object types and instances. However, in many domains, especially those characterized by unstructured or semi-structured textual data, such an OCEL is not readily available. Deriving this detailed information about objects, object types, and activities from unstructured data sources, such as clinical notes, support tickets, or maintenance logs, is significantly more demanding than extracting a traditional event log 10 . The unstructured nature of such data introduces new challenges, such as ambiguous references to objects, contextual or temporal dependencies between actions, domain-specific object types and terminology, and implicit relationships. Consequently, advanced log extraction techniques are essential to bridge this semantic gap between unstructured or semi-structured textual data and the structured OCELs required for OCPM 15 . In this work, we approach this semantic gap by leveraging LLMs within a human-in-the-loop design. 2.2 Case or Object Recognition In conventional XES event log-based process mining, a case represents a single instance of a process, such as a customer order, a patient treatment cycle, or a support ticket, whose events are grouped by a unique case identifier. The case notion is foundational to discovering control-flow models and analyzing execution behavior. However, many real-life systems are not process-aware (e.g., a GPS tracking system, a web traffic log, or industrial devices) and the recorded events lack explicit case identifiers, a problem referred to as elusive cases 22 . Existing approaches to case recognition assume a single case notion and rely on heuristics, rules 2 , or auxiliary data to reconstruct case identifiers 4 ; 26 . Others use process semantics and optimization approaches 1 . However, these methods still misclassify or entirely discard events, especially when only partial information is available or when cases overlap. More importantly, they assume the main object type is known and used as the case notion. To build an object-centric event log from unstructured or semi-structured textual sources, the object types and the number of object types may well be unknown. Additionally, the object identifiers of different types are also unknown and may even be of different formats. For example, in an academic setting, relevant object types may be courses, research projects, conference names, student names, etc.; and for each type the identifiers may have a different format, e.g., course names or even course codes, research paper titles, grant proposal titles, conference names, etc. Identifying relevant object types, their object instances, and correctly associating events with these instances becomes particularly challenging. Recent advances in LLMs present a novel opportunity: LLMs can extract semantic-aware entities from free text 6 , enabling more adaptive and context-aware case recognition. However, their outputs can be inconsistent, especially in domain-specific contexts 20 . This underscores the need for expert-in-the-loop approaches, combining LLM-generated suggestions with domain knowledge to iteratively refine object types, object instances, and activity assignments . 2.3 Semantic Recognition Using LLMs Previous studies have explored activity recognition from natural language text. In 16 , the authors presented a method

Read the complete text