EBuddy: a workflow orchestrator for industrial human–machine collaboration
Michele Banfi, Rocco Felici, Stefano Baraldo, Oliver Avram, Anna Valente · 2026
Read EBuddy: a workflow orchestrator for industrial human–machine collaboration by Michele Banfi, Rocco Felici, Stefano Baraldo, Oliver Avram, Anna Valente (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28579.txt.
Introduction
Banfi and co-authors present EBuddy, a voice-guided workflow tool for human and machine teamwork in factories. Expert know-how is effective but hard to scale, and quality drops when steps are rebuilt ad hoc. EBuddy turns expert practice into a finite state machine. The state gives a clear frame for each spoken request and the allowed next actions. It coordinates GUI software and a collaborative robot by voice. A pilot on impeller blade inspection and repair preparation cut process time a lot.
Catalog record
| CREATOR | Michele Banfi, Rocco Felici, Stefano Baraldo, Oliver Avram, Anna Valente |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33016 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28579
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EBuddy: a workflow orchestrator for industrial human–machine collaboration Michele Banfi, Rocco Felici, Stefano Baraldo, Oliver Avram, Anna Valente Abstract This paper presents EBuddy , a voice-guided workflow orchestrator for natural human–machine collaboration in industrial environments. EBuddy targets a recurrent bottleneck in tool-intensive workflows: expert know-how is effective but difficult to scale, and execution quality degrades when procedures are reconstructed ad hoc across operators and sessions. EBuddy operationalizes expert practice as a finite state machine (FSM) driven application that provides an interpretable decision frame at runtime (current state and admissible actions), so that spoken requests are interpreted within state-grounded constraints, while the system executes and monitors the corresponding tool interactions. Through modular workflow artifacts, EBuddy coordinates heterogeneous resources, including GUI-driven software and a collaborative robot, leveraging fully voice-based interaction through automatic speech recognition and intent understanding. An industrial pilot on impeller blade inspection and repair preparation for directed energy deposition (DED), realized by human-robot collaboration, shows substantial reductions in end-to-end process duration across onboarding, 3D scanning and processing, and repair program generation, while preserving repeatability and low operator burden. 1 Introduction Industrial workcells increasingly combine heterogeneous digital tools (e.g., scanning and inspection software, data-processing pipelines, robot controllers) with collaborative automation. In practice, performance is often constrained less by the robot’s physical capability than by the workflow decision space: knowing what step is valid next, choosing parameters, handling exceptions, and coordinating handovers between software and hardware. These choices are usually clear to a small number of domain experts, yet they are hard to operationalize consistently across operators, shifts, and product variants. This paper presents EBuddy, a voice-guided workflow orchestrator that represents procedures as finite state machine (FSM) recipes and uses the current state to frame and constrain operator commands. Interaction becomes a sequence of admissible decisions: the operator always has a stable view of the active state and available actions, and can progress through state-aware voice commands; workflows remain extensible via modular and nested FSMs without becoming unreadable. The approach is demonstrated in an industrial pilot on a turbomachinery impeller inspection and repair preparation pipeline, spanning GUI-driven scanning software, 3D data processing, and cobot-supported operations with quantitative and qualitative evidence indicating improved repeatability and reduced execution friction in a multi-tool setting. 2 Related Works 2.1 Voice-based human-machine interaction Voice-based human–machine interaction has evolved from early keyword-based speech interfaces to context-aware, conversational systems, driven largely by deep learning advances in automatic speech recognition, language understanding, and speech synthesis [ 15 , 10 ] . Contemporary speech user interfaces now underpin applications ranging from mobile digital voice assistants and smart speakers to social robots. Voice-based interaction is emerging as a key modality also in industrial settings, enabling operators to issue hands-free commands while maintaining situational awareness and manual dexterity on the shop floor. Recent work on collaborative robots shows that Automatic Speech Recognition (ASR) can achieve reliable speech recognition in noisy environments, even in presence of technical vocabulary [ 4 ] . 2.2 Workflow orchestration in industriay and HRC In the last years, foundation models have demonstrated remarkable generalization capacities across tasks, and the recent introduction of standard protocols for agent-agent and agent-tool interaction [ 2 ] opened several new opportunities for integration with heterogeneous workflows. Yet, their direct deployment in manufacturing and industrial robotics remains constrained by a lack of contextual awareness, insufficient interpretability, and rigid integration with proprietary industrial software ecosystems. Consequently, researchers have increasingly focused on hybrid approaches that combine AI reasoning with symbolic, procedural, or knowledge-based control to ensure predictable behavior and compliance with safety and quality standards [ 6 , 16 , 13 ] . Systems such as AI-enhanced CAD-CAM environments, context-aware programming assistants, and conversational interfaces for robotic programming aim to lower the cognitive and technical barriers to advanced production workflows [ 8 , 14 , 1 ] . Still, many human-robot collaboration (HRC) systems assume skilled users, thus raising a barrier to broader adoption. Moreover, human–AI collaboration lies at the core of the emerging Industry 5.0 paradigm, as human agency and oversight are considered fundamental to guarantee the safety and transparency of human-robot collaborative systems [ 5 ] . Within this domain, studies emphasize technologies that augment, rather than replace, human expertise [ 12 ] . These findings support the design of EBuddy as an explicit workflow orchestration partner, rather than a passive automation tool, that executes, monitors, and explains complex workflows while remaining accountable to the human operator. 3 Methodology This work addresses a fundamental challenge: users at different expertise levels often execute different workflows for the same industrial process, which creates both knowledge transfer gaps and process variability. Experts consistently reach high-quality outcomes by combining domain know-how with fluent tool use; intermediate users only partially master the problem and the toolchain and therefore deviate more often; beginners typically require sustained tutoring, tying up expert time that could be spent on higher-value work. EBuddy frames human–machine collaboration as a shared workflow orchestration framework, in which expert knowledge is explicitly encoded and operationalized through FSM (Finite-State-Machine) based workflows, multimodal interfaces, and distributed computation. At runtime, the FSM provides an interpretable decision context, namely the current state and admissible actions, so operators can make sequential choices that remain consistent with process logic. By separating core orchestration from domain-specific artifacts (e.g., JSON-defined workflows, Python control scripts, instructional slides), the system enables asynchronous human–human knowledge transfer and progressive automation without removing expert oversight. The overall system architecture, described in Fig. 1 , is composed by the following components: 1. Role-based interaction environment: manages user responsibilities (Sec. 3.1 ). 2. FSM architecture: implements different Workflows (Sec. 3.2 ). 3. Tool controls : allow to control UI-based applications and physical tools such as robots (Sec. 3.3 ). 4. User interface (UI): informs users of process state, available commands, and context-sensitive guidance according to workflow progression (Sec. 3.4 ). 5. Language understanding pipeline, composed by ASR and context-aware intent recognition (Sec. 3.5 ). Figure 1: Collaboration between users and EBuddy to achieve optimal tool interactions, guided by domain expert knowledge encoded within EBuddy’s workflow framework. 3.1 Knowledge transfer and user roles EBuddy transfers application workflow knowledge from an expert—who defines workflows (FSMs with states and transitions), Tool controls, and contextual documentation—to technicians with different skill levels through a role-based interaction environment. Fig. 1 demonstrates the different interactions with EBuddy of an advanced user and of a beginner/intermediate user. In this light, EBuddy distinguishes three user profiles with progressively shifting responsibilities. Beginners rely on step-by-step guidance and mainly confirm the actions proposed by the default workflow: they observe EBuddy’s execution to learn core procedures, perform necessary physical interventions, and acknowledge safety-critical prompts. Intermediate users still use EBuddy as the primary interface but increasingly take local decisions, such as selecting among predefined options and tuning a limited set of parameters; they may enable automation for routine segments, handle common warnings, and monitor output quality to decide whether additional steps are needed. Advanced users act as process authors and maintainers: they formalize expert know-how by structuring the overall process into workflows and tool interaction primitives (widgets, scripts, checks, and rollback actions). They also establish defaults and safety guardrails, validate and refine execution strategies, and curate the instructional material that supports less experienced users. As proficiency increases,