Interactive Robot Programming for Surface Finishing via Task-Centric Mixed Reality Interfaces

Christoph Willibald, Lugh Martensen, Thomas Eiband, Dongheui Lee · 2025

Read Interactive Robot Programming for Surface Finishing via Task-Centric Mixed Reality Interfaces by Christoph Willibald, Lugh Martensen, Thomas Eiband, Dongheui Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-23616.txt.

Introduction

Christoph Willibald, Lugh Martensen, Thomas Eiband, and Dongheui Lee want factory robots programmed by the people who use them, not only by robotics experts. Long, expert-heavy setup is why collaborative robots are rarely used for surface finishing in small workshops, where products vary and batches are small. Their answer is a mixed reality interface organized around the finishing task itself.

Through the interface, a user marks regions of a workpiece for processing, guided by a new segmentation algorithm that folds human input into its surface model. Continuous visual feedback lets the user refine the segmentation, after which the robot's path over the chosen area is generated automatically. In two user studies, the chosen design cut workload and improved usability, letting even inexperienced users program finishing tasks effectively.

Catalog record

CREATORChristoph Willibald, Lugh Martensen, Thomas Eiband, Dongheui Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-23616

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interactive Robot Programming for Surface Finishing via Task-Centric Mixed Reality Interfaces Christoph Willibald, Lugh Martensen, Thomas Eiband, Dongheui Lee Abstract Lengthy setup processes that require robotics expertise remain a major barrier to deploying robots for tasks involving high product variability and small batch sizes. As a result, collaborative robots, despite their advanced sensing and control capabilities, are rarely used for surface finishing in small-scale craft and manufacturing settings. To address this gap, we propose a novel robot programming approach that enables non-experts to intuitively program robots through interactive, task-focused workflows. For that, we developed a new surface segmentation algorithm that incorporates human input to identify and refine workpiece regions for processing. Throughout the programming process, users receive continuous visual feedback on the robot’s learned model, enabling them to iteratively refine the segmentation result. Based on the segmented surface model, a robot trajectory is generated to cover the desired processing area. We evaluated multiple interaction designs across two comprehensive user studies to derive an optimal interface that significantly reduces user workload, improves usability and enables effective task programming even for users with limited practical experience. I Introduction Especially repetitive and physically intensive tasks are not attracting enough workers in today’s competitive job market anymore [ 6 ] . Robotic solutions have the potential to alleviate the labour shortage problem by taking over unattractive tasks, while workers can focus on supervision and instruction. This provides a safe, ergonomic, and attractive work environment for the upcoming generations while making each workplace more productive. Fig. 1 : Our proposed human-robot interface for programming surface finishing tasks. The task-centric approach allows users to focus on the task to perform on the workpiece, instead of programming the robot’s motions step-by-step. To be economically viable in the context of small and medium-sized enterprises (SMEs), robotic solutions need to be highly re-configurable in minimal time to accommodate a variety of tasks [ 21 ] . One particularly impactful yet challenging area for automation in small craft businesses is surface processing, including tasks such as sanding and polishing. These tasks are difficult to implement due to the significant variability in workpiece shapes and dimensions. Since collaborative robots have the required sensors and control algorithms to operate safely with humans in the same workspace and to reliably perform in-contact motions, they pave the way for such applications. However, the lack of flexible and intuitive software solutions for robot programming prevents SMEs from integrating collaborative robots into their workflow. To improve this situation, active research in human-machine interface design and learning from demonstration aims to enable non-experts to naturally interact with robots to teach them new tasks [ 18 , 8 , 31 ] . Regardless of the essential role that humans play in this process, little focus has been put on measuring and improving the teacher’s ability to provide good task instructions. Howard and Sena [ 26 ] show that there is a discrepancy between the user’s belief of the robot’s knowledge and its actual capabilities if the user receives insufficient feedback from the robot during the teaching process. This knowledge gap leads to a teaching performance degradation. As shown in Fig. 1 , this work investigate the use of Mixed Reality (MR)-based human-robot interfaces to close the knowledge gap and to facilitate intuitive programming of surface finishing tasks. Once the environment point cloud is loaded, the user selects the target area for processing and receives continuous visual feedback on the predicted surface model. The prediction can be adjusted and refined through additional user input until it aligns with the intended processing area. We introduce an algorithm that combines user input with a 3D point cloud to fit a surface model to the selected area (steps 2 and 3 in Fig. 1 ). The user can subsequently edit the surface model and virtually validate the robot’s finishing strategy before deploying it on the hardware. The contributions of this work are twofold • We developed a novel surface segmentation algorithm that uses an environment point cloud and user-demonstrated contact points to infer the shape primitive and the area of a workpiece to process. This enables the robot to utilize the optimal strategy for finishing the identified surface segment while eliminating the need to model the geometry of the workpiece. • We conducted comprehensive user studies to evaluate the usability and effectiveness of various human-robot interface concepts for programming of surface finishing tasks, leading to the derivation of optimal interaction patterns. II Related Work Robotic surface finishing typically begins with either locating the geometry of a known workpiece or generating a surface model for processing when no prior geometry information is available. In the latter case, several approaches [ 32 , 15 , 30 , 1 ] rely on obtaining a high-resolution point cloud of the workpiece, followed by statistical outlier filtering to eliminate scanning artifacts. Once the point cloud is refined, it is uniformly sliced by parallel intersection planes to generate a trajectory that covers the entire surface [ 32 , 15 , 30 ] or to remove surface defects that were identified using curve fitting and outlier detection [ 1 ] . The points on these intersection planes, along with their surface normals, serve as via points for the robot’s end-effector pose during the surface finishing process. To optimize these via points, Liang et al. [ 15 ] apply the Douglas-Peucker algorithm [ 5 ] , which reduces point density in areas of low curvature. Wang et al. [ 30 ] employ Non-Uniform Rational B-Spline (NURBS) curve fitting, ensuring smooth tool paths and consistent tool orientations throughout the trajectory. Xia et al. [ 34 ] adopt a haptic exploration scheme to identify surface geometry and stiffness, where an operator guides the robot along the object’s surface while the robot maintains a constant contact force. The described approaches for unknown workpiece geometry still require great effort in manual preprocessing for the user. We reduce this effort by utilizing a model-fitting algorithm that is robust against outliers. For scenarios with known workpiece geometries, several approaches [ 16 , 27 , 17 , 24 , 35 ] utilize CAD models for surface segmentation and coverage planning. Liu et al. [ 16 ] partition the CAD model into flat and curved segments, while [ 27 ] use surface normals from tessellated CAD models to divide the surface into flat segments with different normal vector directions. In [ 17 ] , the CAD model is uniformly sampled to generate a point cloud. Feature vectors comprising the point coordinates and their normal vectors are then used to segment the surface through k-means clustering. Schneyer et al. [ 24 ] consider both tool and workpiece geometry to identify segments that can be processed with the same strategy. Further, they adapt tool trajectories based on the contact area between the tool and the workpiece. Similarly, Wen et al. [ 33 ] use a contact model incorporating applied normal force and local workpiece curvature to generate trajectories with adaptive spacing. In [ 36 ] , the point clouds of a finished and an unfinished workpiece are aligned with iterative closest point matching to detect uneven surfaces and to generate a sanding trajectory for their removal. In contrast to these

Read the complete text