THÖR-MAGNI Act : Actions for Human Motion Modeling in Robot-Shared Industrial Spaces

Tiago Rodrigues de Almeida, Tim Schreiter, Andrey Rudenko, Luigi Palmieri, Johannes A. Stork, Achim J. Lilienthal · 2024

Read THÖR-MAGNI Act : Actions for Human Motion Modeling in Robot-Shared Industrial Spaces by Tiago Rodrigues de Almeida, Tim Schreiter, Andrey Rudenko, Luigi Palmieri, Johannes A. Stork, Achim J. Lilienthal (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13729.txt.

Introduction

Tiago Rodrigues de Almeida and co-authors present a dataset paper, printed in 2024, recording what people actually do around mobile robots in industrial halls. Predicting human motion is essential for safe robot coworkers, yet fine grained action labels in factory settings barely exist. The team extended an existing motion dataset with more than eight hours of hand labeled actions taken from eye tracking glasses worn by participants.

The actions follow a long tail of everyday moves, paired with rich profiles of speed and travel distance. Two demonstration tasks show the data at work: forecasting a path given the action, and forecasting both together, where the authors' transformer models beat the baselines. The dataset aims to sharpen robot awareness on the factory floor.

Catalog record

CREATORTiago Rodrigues de Almeida, Tim Schreiter, Andrey Rudenko, Luigi Palmieri, Johannes A. Stork, Achim J. Lilienthal
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-13729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

THÖR-MAGNI Act : Actions for Human Motion Modeling in Robot-Shared Industrial Spaces Thanks: This work was supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) and by the EU Horizon 2020 No. 101017274 (DARKO). Tiago Rodrigues de Almeida, Tim Schreiter, Andrey Rudenko, Luigi Palmieri, Johannes A. Stork, Achim J. Lilienthal Abstract Accurate human activity and trajectory prediction are crucial for ensuring safe and reliable human-robot interactions in dynamic environments, such as industrial settings, with mobile robots. Datasets with fine-grained action labels for moving people in industrial environments with mobile robots are scarce, as most existing datasets focus on social navigation in public spaces. This paper introduces the THÖR-MAGNI Act dataset, a substantial extension of the THÖR-MAGNI dataset, which captures participant movements alongside robots in diverse semantic and spatial contexts. THÖR-MAGNI Act provides 8.3 hours of manually labeled participant actions derived from egocentric videos recorded via eye-tracking glasses. These actions, aligned with the provided THÖR-MAGNI motion cues, follow a long-tailed distribution with diversified acceleration, velocity, and navigation distance profiles. We demonstrate the utility of THÖR-MAGNI Act for two tasks: action-conditioned trajectory prediction and joint action and trajectory prediction. We propose two efficient transformer-based models that outperform the baselines to address these tasks. These results underscore the potential of THÖR-MAGNI Act to develop predictive models for enhanced human-robot interaction in complex environments. I Introduction Human movement and actions are shaped by many factors that collectively define the context in which individuals navigate and interact [ 1 ] . These factors may be internal, such as the person’s own tasks, goals, and preferences, as well as external, i.e., coming from the environment, such as the location of obstacles, affordances, and semantically meaningful regions [ 2 ] . Robots can detect external factors and infer certain internal factors, using them to anticipate future motion and activities [ 3 , 4 , 5 ] to predictively assist and navigate safely and efficiently [ 6 ] . Especially in industrial environments, robots face complex yet structured activities of people and interactions with other workers and robots [ 7 ] . Datasets containing accurate labels of human motion in such environments are rare, with most works focusing on social navigation in public spaces where the dominant activity is walking and standing [ 8 , 9 , 10 , 11 ] . The recent THÖR-MAGNI dataset aims to address this gap by providing a large-scale indoor motion capture recording of human navigation and robot interaction [ 12 ] . It encompasses five distinct scenarios that simulate typical activities in industrial environments, such as transportation of various objects, interaction with robots, and goal-oriented navigation alone and in groups. THÖR-MAGNI includes 3.5 hours of motion from 40 participants and 8.3 hours of egocentric videos from 16 participants over five recording days. The activity labels in THÖR-MAGNI are, however, limited to the invariable role of each participant (referred to as agent class), representing a complex activity assigned to the participant for the duration of the experiment. These roles have been shown to improve trajectory prediction [ 13 ] and describe gaze behavior [ 14 ] , but fine-grained action labels would be required to model the separate sub-tasks and their durations in each activity. Fig. 1: Action annotations for a 4-minute recording of a person carrying storage bins while interacting with a mobile robot, synchronized with the motion capture data. Inset images display snapshots from gaze overlaid videos, featuring visualizations of head orientation vector ( red ) and gaze vector ( green ). The length of the arrows on the map denotes the velocity magnitude. In this paper, we present THÖR-MAGNI Act – an extension of the original dataset, which provides 8.3 hours of fine-grained actions derived from the first-person view videos of participants wearing eye-tracking glasses. Our THÖR-MAGNI Act is unique in aligning action labels with high-quality multi-modal first-person gaze and third-person motion capture data, as shown in Fig. 1 . These labels enable the robot to anticipate long-term human motion trajectories and actions. To demonstrate the utility of THÖR-MAGNI Act , we present two transformer-based trajectory prediction frameworks: (1) role- and action-conditioned trajectory prediction, extending [ 13 ] , and (2) joint prediction of future trajectory and corresponding actions via multi-task learning [ 15 ] . Our results show how incorporating action labels can improve the performance of these predictive models. THÖR-MAGNI Act and the corresponding scripts are stored in a publicly accessible repository 1 1 1 https://github.com/tmralmeida/thor-magni-actions . Documentation on how to use and visualize the dataset can be found in the same repository. II THÖR-MAGNI Act II-A Experimental Design The THÖR-MAGNI data acquisition, conducted in a laboratory, simulates industrial logistics settings to explore diverse human-human and human-robot interactions [ 12 ] . The dataset comprises five scenarios, distinguished by the spatial layout, the mode of the robot operation, and specific tasks assigned to the participants. These tasks or agent classes represent the high-level activities assigned to the person for the duration of a 4-minute recording session, such as: Carrier–Box , Carrier–Bucket , Carrier–Large Object , Visitors–Alone , Visitors–Group , and Visitors–Alone HRI . In each session, one to three participants wear eye-tracking glasses that capture egocentric video data. In [ 12 ] , we provided standardized instructions to ensure natural behavior, informing participants that the experiment aimed to evaluate the robot’s perception of human actions. In particular, participants were asked to carry stacks of boxes and buckets between the source and target locations ( Carrier–Box and Bucket ). Some participants moved a large poster stand ( Large Object ) in pairs of two. Visitors navigated freely between goal points, drawing a random card to determine their next destination. Visitor–Alone HRI involved both passive interactions (passing the robot in close proximity) and active interactions (joint navigation to goal points) with a mobile robot. The Carrier–Storage Bin HRI collaboratively with the robot transported storage bins between random goal points. II-B Action Annotations In this work, we define a set \mathcal{A} of 14 unique action labels: Walk , DrawCard , observing another person drawing a card at a goal point ( ObserveCardDraw ), moving a larger object ( WalkLO ), PickBucket , WalkBucket , DeliverBucket , PickBox , WalkBox , DeliverBox , PickStorageBin , WalkStorageBin , DeliverStorageBin , and HRI , based on the existing agent classes in THÖR-MAGNI. Each agent class is associated with specific actions, while some actions are shared across different agent classes (see Fig. 2 for an overview). An agent class is constant in all trajectories of a particular agent, whereas an action class may change at every time step. Consequently, this data extension provides finer labeling of internal factors (e.g., goal-driven actions) that can influence human motion. In particular, the action classes Walk and DrawCard are shared across multiple agent classes, indicating that trajectories involving these actions are likely to have similar characteristics, even when performed by agents of different agent classes. We annotated these actions in the entire THÖR-MAGNI dataset using the 8.3 hours of egocentric videos and the “Event Marker” feature in the eye-tracking software [ 16 ] . Markers were placed at initial fixations indicating action switches, such as reaching for objects or bending to deliver items. For ambiguous switches

Read the complete text