Graphormer-Guided Task Planning: Beyond Static Rules with LLM Safety Perception

Wanjing Huang, Tongjie Pan, Yalan Ye · 2025

Read Graphormer-Guided Task Planning: Beyond Static Rules with LLM Safety Perception by Wanjing Huang, Tongjie Pan, Yalan Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06866.txt.

Introduction

Wanjing Huang, Tongjie Pan, and Yalan Ye propose a safety aware task planning frame for robots in a 2025 paper. Large language models now generate robot task sequences well, but safe execution lags behind. Fixed safety rules cannot adapt to hazards that arise live, and structured risk perception is weak in current methods, which matters for low latency, safety critical work.

Their frame pairs model based decisions with a dynamic spatio semantic safety graph that captures space and context risk. A risk perception module updates safety predictions during execution, so plans refine as hazards appear. Experiments in a simulated home environment show gains in risk detection accuracy, safety notice, and task adaptability over static rule methods.

Catalog record

CREATORWanjing Huang, Tongjie Pan, Yalan Ye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06866

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Graphormer-Guided Task Planning: Beyond Static Rules with LLM Safety Perception Wanjing Huang, Tongjie Pan, Yalan Ye Abstract Recent advancements in large language models (LLMs) have expanded their role in robotic task planning. However, while LLMs have been explored for generating feasible task sequences, their ability to ensure safe task execution remains underdeveloped. Existing methods struggle with structured risk perception, making them inadequate for safety-critical applications where low-latency hazard adaptation is required. To address this limitation, we propose a Graphormer-enhanced risk-aware task planning framework that combines LLM-based decision-making with structured safety modeling. Our approach constructs a dynamic spatio-semantic safety graph, capturing spatial and contextual risk factors to enable online hazard detection and adaptive task refinement. Unlike existing methods that rely on predefined safety constraints, our framework introduces a context-aware risk perception module that continuously refines safety predictions based on real-time task execution. This enables a more flexible and scalable approach to robotic planning, allowing for adaptive safety compliance beyond static rules. To validate our framework, we conduct experiments in the AI2-THOR environment. The experiments results validates improvements in risk detection accuracy, rising safety notice, and task adaptability of our framework in continuous environments compared to static rule-based and LLM-only baselines. Our project is available at https://github.com/hwj20/GGTP I INTRODUCTION A household robot retrieving vegetables from the refrigerator may entirely fail to notice a child approaching a hot stove. Existing robotic task planning frameworks, including rule-based systems and large language model (LLM)-driven planners, focus exclusively on predefined task sequences while lacking lwo-latency environmental risk perception [ 1 ] . These systems evaluate task execution based on internal constraints, ensuring rule compliance but ignoring external hazards. For instance, a rule-based model may enforce that a knife is only used on a cutting board but fails to recognize a child reaching for it as a risk. applications. While LLMs enable flexible, generalizable task reasoning [ 2 , 3 ] , they lack structured risk perception, making them unsuitable for safety-critical Reinforcement learning (RL) approaches [ 4 ] can optimize decision-making in controlled settings but require extensive training and fail to generalize in low-latency scenarios [ 2 ] . We summarize three key limiations in current approaches to LLM-driven robotic task planning [ 2 , 5 ] : • Lack of structured risk perception: Most LLM-based planners operate purely on semantic reasoning, failing to incorporate spatially structured risk modeling. As a result, they struggle to detect high-risk interactions, such as a child approaching a hazardous object. • Static rule-based safety mechanisms: Traditional rule-based safety approaches require manually predefined constraints, making them rigid and difficult to scale. • Limited task adaptability: Existing frameworks typically generate task sequences in a feedforward manner, with limited capacity to modify execution plans in response to safety emergencies. To address these challenges, we propose a Graphormer-enhanced risk-aware task planning framework that integrates structured safety perception with LLM-driven decision making. We compare with the static rule-based method in Fig. 2 . Our work is inspired by the paradigm shift introduced by graph neural networks (GNNs) in autonomous driving, where structured reasoning has proven instrumental in risk prediction and trajectory optimization [ 6 , 7 , 8 , 9 ] . GNN architectures have been extensively applied to model spatial-temporal interactions, facilitating predictive control by encoding agent-object dependencies within a graph-based framework. These approaches [ 9 , 8 ] employ hierarchical attention mechanisms to selectively prioritize critical relational structures, thereby refining motion forecasting and enhancing decision robustness in unstructured scenarios. Building upon the above principles, our approach adaptively constructs a spatio-semantic safety graph, encoding both spatial and contextual risk attributes to elevate real-time hazard perception. The Graphormer [ 10 ] model is designed to emphasize high-risk interactions through an attention-weighted graph representation, enabling continuous risk assessment and adaptive task refinement. During execution, the system iteratively updates its risk evaluation, ensuring that, upon the emergence of a hazardous condition, a task replanning mechanism is invoked. Using LLM-based semantic reasoning in conjunction with structured risk modeling, our framework synthesizes online task adaptation with proactive safety measures, significantly improving robustness in open-world robotic applications. Fig. 1: Comparison between the static rule-based method and our approach. LTL-based safety enforcement relies on predefined constraints, which inherently lack spatial perception and fail to capture unenumerated risks. As a result, when a robot executes a task such as retrieving ingredients from the refrigerator, it is unable to recognize a child approaching a high-temperature oven. Consequently, LTL transmits an incorrect ”safe” signal to the LLM planner, leading to a failure in proactive hazard avoidance. In contrast, our method models spatial-semantic relationships, enabling real-time risk assessment and adaptive task modification. In summary, the main contributions of this study can be outlined as follows: • We introduce a spatio-semantic safety graph that models environmental risk factors based on Graphormer, providing a structured representation of hazards beyond conventional rule-based methods. • We develop an adaptive LLM decision framework that enables adaptive task modifications when detecting risks, with flexible safety compliance for continuous environments. • We validate our framework through extensive experiments in AI2-THOR [ 11 ] , demonstrating substantial improvements in risk detection and adaptive task execution compared to static rule-based and LLM-only baselines. II RELATED WORK TABLE I: Comparison of Related Work and Our Approach Method LLM Planning Safety Model Adaptivity Real-time Risk Perception AiGem [ 6 ] ✗ Graph-based ✓ ✓ SMART-LLM [ 3 ] ✓ ✗ ✗ ✗ Plug in the Safety Chip [ 1 ] ✓ Rule-based ✗ ✗ SafePlanner [ 12 ] ✓ Pretrained Safety Prediction Model ✓ ✗ RoCo [ 13 ] ✓ ✗ ✓ ✗ Co-NavGPT [ 14 ] ✓ ✗ ✓ ✗ TidyBot [ 15 ] ✓ ✗ ✗ ✗ AutoRL [ 16 ] ✓ Human Guardrails ✓ ✗ Cross-Layer Sequence Supervision [ 17 ] ✓ LTL Constraints ✗ ✗ Ours ✓ Graphormer-enhanced ✓ ✓ II-A LLMs in Robotic Task Planning Recently rich literature has tried to employ LLMs and to accomplish comprehensive planning in complex long-horizon robotics tasks [ 18 ] . Studies [ 2 , 5 ] have shown that LLMs can generate high-level action sequences with world knowledge and semantic reasoning. LLM-driven agents have been applied to domestic robotics [ 15 ] , multi-agent collaboration [ 3 , 18 ] , and interactive task execution [ 19 ] . Despite their remarkable capabilities in task reasoning and generalization [ 20 ] , LLMs exhibit several fundamental challenges when applied to robotic planning and control. First, LLM-generated task sequences often suffer from inconsistencies and unpredictability, as minor variations in input prompts can lead to significantly different outputs [ 5 ] . This lack of determinism raises critical safety concerns, particularly in real-world robotic deployments where failure to adhere to strict operational constraints can result in hazardous consequences. Second, existing LLM-based planning frameworks rely heavily on prompt engineering, yet there is no standardized methodology for designing prompts that ensure robust and contextually aware decision-making [ 5 , 21 ] . Without a structured framework for encoding safety constraints, robots may misinterpret task objectives, leading to unsafe execution behaviors. II-B Rule-Based Safety Constraints in Robotics Linear Temporal Logic (LTL) has found utility innforcing formal safety constraints in robotic planning and control [ 22 , 17 , 23

Read the complete text