GNN-based Decentralized Perception in Multirobot Systems for Predicting Worker Actions

Ali Imran, Giovanni Beltrame, David St-Onge · 2025

Read GNN-based Decentralized Perception in Multirobot Systems for Predicting Worker Actions by Ali Imran, Giovanni Beltrame, David St-Onge (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04193.txt.

Introduction

Ali Imran, Giovanni Beltrame, and David St-Onge present a 2025 system that lets robot teams predict what a human worker will do next. In shared work spaces, such forecasts keep people safe and work smooth. Each robot in their system builds a graph map of its surroundings and shares it with the team. No central brain runs the setup.

Shared maps join with time records to track how a worker moves and acts. A graph neural network reads the joined data, and a swarm-style vote makes all robots settle on one reading of the act. Tests showed that more robots and longer time spans both raised forecast accuracy. The vote step also made the team harder to throw off in busy scenes. Decentralized teams could suit factories where one control point is a weak link.

Catalog record

CREATORAli Imran, Giovanni Beltrame, David St-Onge
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-04193

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GNN-based Decentralized Perception in Multirobot Systems for Predicting Worker Actions Ali Imran, Giovanni Beltrame, David St-Onge Abstract In industrial environments, predicting human actions is essential for ensuring safe and effective collaboration between humans and robots. This paper introduces a perception framework that enables mobile robots to understand and share information about human actions in a decentralized way. The framework first allows each robot to build a spatial graph representing its surroundings, which it then shares with other robots. This shared spatial data is combined with temporal information to track human behavior over time. A swarm-inspired decision-making process is used to ensure all robots agree on a unified interpretation of the human’s actions. Results show that adding more robots and incorporating longer time sequences improve prediction accuracy. Additionally, the consensus mechanism increases system resilience, making the multi-robot setup more reliable in dynamic industrial settings. I Introduction Collaborative robots are poised to become a cornerstone of Industry 5.0 [ 1 ] , emphasizing human-centric design solutions to meet the flexibility demands of hyper-customized industrial processes [ 2 ] . Significant efforts have been directed toward identifying key enabling technologies to enhance robotic systems with advanced situational awareness and robust safety features for human coworkers. Two pivotal technologies stand out: individualized human-machine interaction systems that merge the strengths of humans and machines, and the application of AI to improve workplace safety [ 3 ] . In highly collaborative and hazardous scenarios such as manufacturing facilities, robots must develop a holistic understanding of their environment to ensure efficiency and safety. While humans excel at anticipating events due to their spatial reasoning, understanding of others’ behavior, and planning under uncertainty [ 4 ] , robotic systems lack such innate capabilities. However, they can leverage distributed sensing networks and algorithms to model spatial and temporal relationships within a scene. Fig. 1 : Overview of the multi-robot system deployed in Isaac Sim for human intent prediction. Robots generate spatial graph representations, integrate neighbor information, employ RNNs for temporal understanding, and converge using a swarm-intelligence-inspired consensus mechanism Fig. 2 : System architecture for multi-robot human intent prediction: a) robot detects the human and objects of interest in the scene, feature vectors are created through an encoder. b) Graph representation of the scene is created. c) Intent prediction based on GNN is made. This information is shared with other robots and the Temporal block. d) Human pose keypoints and spatial information from other robots are aggregated and passed through the temporal understanding block to make ego and collective intent predictions. This information is again shared with other robots. e) Concensus mechanism utilizes the predictions and quality of visual information from the robot to converge to a single decision about the intent. Algorithms for predicting human behavior and future actions have been explored across domains such as pedestrian crossing prediction [ 5 , 6 , 7 ] , intent interpretation in assistive robotics [ 8 , 9 ] , and forecasting human poses in collaborative workspaces [ 10 , 11 ] . Most approaches rely on human detection and tracking, using local features like pose, velocity, and location [ 12 , 13 , 14 , 15 ] . However, these methods often overlook the relationships between humans and surrounding objects, leading to poor performance when encountering unfamiliar human behavior and limiting their ability to anticipate further into the future. Recent advancements in graph-based models show good potential for spatiotemporal intent prediction [ 16 ] . They demonstrate the potential of graph formalisms to emphasize relationships between entities rather than their individual properties, offering generalizable insights [ 17 ] . In parallel, Graph Neural Networks (GNNs) have gained prominence in applications such as multi-robot path planning [ 18 ] , collaborative perception [ 19 ] , and navigation in complex environments [ 20 ] . Building on this foundation, we extend GNN-based approaches to scenarios with a broader range of human actions, leveraging multi-robot systems to develop a shared understanding of the environment. Multi-robot systems provide significant advantages over single-robot systems, including enhanced robustness, scalability, and flexibility [ 21 ] . While many implementations rely on centralized control, a decentralized approach offers additional benefits, such as improved system flexibility and fault tolerance [ 22 ] . Decentralization, however, presents its own challenges, particularly in achieving effective coordination and collective decision-making among robots [ 23 ] . These challenges arise because, in the absence of a central controller, robots must rely on distributed algorithms to interpret their environment and align their actions. Consensus mechanisms in swarm intelligence have been widely explored to address these issues. In decentralized multi-robot systems, such as those used in swarm robotics, consensus ensures that robots can collectively make coherent decisions even with diverse inputs and local observations. In decentralized multi-robot systems, such as swarms of robots, consensus can be achieved using strategies like majority voting [ 24 ] , ranked voting systems [ 25 ] , and entropy-based local negotiation [ 26 ] . These methods enable efficient decision alignment while accommodating sensor variability and differences in information quality. Our shared perception intent prediction pipeline employs graph-based methods to facilitate information exchange between robots. Implemented in ROS, the pipeline integrates data from multiple robots to model spatial relationships between humans and nearby objects using GNNs and temporal relations with Recurrent Neural Networks (RNNs), enabling accurate human intent prediction. This multi-robot strategy enhances robustness by compensating for individual sensor failures and ensuring safety in industrial environments. The main contributions of this paper are as follows: • Development of a spatial understanding module leveraging graph neural networks to model relationships between humans and nearby objects. • Introduction of a temporal understanding module that aggregates spatial data from other robots in a decentralized manner to predict and forecast human actions. • Integration of a swarm intelligence-inspired consensus mechanism to ensure decision convergence across all robots in the system. II Method II-A Problem statement In manufacturing setups, numerous dynamic processes such as machining, assembly, and material handling occur simultaneously. Human workers perform multiple tasks like machine tending and assembly, requiring adaptability. These dynamic conditions pose significant challenges for robots, which must operate efficiently while prioritizing human safety. Although manufacturing represents just one subset of potential robot deployment scenarios, it serves as a baseline and motivation for designing our multi-robot system. We define intent prediction as the anticipation of a human operator’s future action within a specific time horizon. For this study, we consider time horizons of 1s, 2s, and 3s into the future, based on the Stopping time and distance metric outlined in ISO 10218-1:2011 safety standards, Annex B(normative) [ 27 ] . This metric relates to the time required for a robot to detect a human, initiate deceleration, and come to a complete stop, which establishes the minimum threshold for the perception pipeline’s prediction capabilities. Given the relatively low speeds of industrial mobile robots 1 1 1 MiR250 [ 28 ] and Otto 1500 [ 29 ] both have a top speed of 2.0 m/s , the selected time horizons are deemed sufficient for safety and operational efficiency. For instance, if a human is detected at a distance of 8 meters while both the robot and the human are moving

Read the complete text