Human-Aided Trajectory Planning for Automated Vehicles through Teleoperation and Arbitration Graphs

Nick Le Large, David Brecht, Willi Poh, Jan-Hendrik Pauls, Martin Lauer, Frank Diermeyer · 2025

Read Human-Aided Trajectory Planning for Automated Vehicles through Teleoperation and Arbitration Graphs by Nick Le Large, David Brecht, Willi Poh, Jan-Hendrik Pauls, Martin Lauer, Frank Diermeyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-02207.txt.

Introduction

Nick Le Large and co-authors add human teleoperation aid to automated vehicle planning through arbitration graphs. Self-driving systems stall in scenes they cannot solve alone. Remote human help can extend their safe operating range.

Assistance concepts let operators nudge single modules like planning. Deep integration makes such concepts hard to build and maintain. Arbitration graphs offer a modular decision frame instead. The team inserted remote aid without changing core software parts. Operators could adjust planner limits in live simulations. They could also unlock paths beyond normal operating bounds. Two use cases proved the approach in simulation. The design eases adding human backup to fleets in hard scenes.

Catalog record

CREATORNick Le Large, David Brecht, Willi Poh, Jan-Hendrik Pauls, Martin Lauer, Frank Diermeyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-02207

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human-Aided Trajectory Planning for Automated Vehicles through Teleoperation and Arbitration Graphs Nick Le Large, David Brecht, Willi Poh, Jan-Hendrik Pauls, Martin Lauer, Frank Diermeyer Abstract Teleoperation enables remote human support of automated vehicles in scenarios where the automation is not able to find an appropriate solution. Remote assistance concepts, where operators provide discrete inputs to aid specific automation modules like planning, is gaining interest due to its reduced workload on the human remote operator and improved safety. However, these concepts are challenging to implement and maintain due to their deep integration and interaction with the automated driving system. In this paper, we propose a solution to facilitate the implementation of remote assistance concepts that intervene on planning level and extend the operational design domain of the vehicle at runtime. Using arbitration graphs, a modular decision-making framework, we integrate remote assistance into an existing automated driving system without modifying the original software components. Our simulative implementation demonstrates this approach in two use cases, allowing operators to adjust planner constraints and enable trajectory generation beyond nominal operational design domains. \svgpath figures/ I Introduction Level 4 automated vehicles (AVs) have been deployed progressively in commercial and research settings in recent years. However, AVs still encounter situations that cannot be solved by automation alone, leading to disengagements, i.e. deactivation of the automated driving system (ADS) that prevent mission completion. An exemplary disengagement scenario is depicted in Fig. 1 where a object is blocking the AV ’s lane, forcing it to stop due to the inability to proceed without human support to assess and resolve the scenario. To address these situations, AV companies use teleoperation technology, enabling human remote operators (ROs) to assist AVs remotely via mobile networks [ 1 , 2 , 3 ] . For example, Cruise required approximately 1.5 workers per vehicle, including remote assistance personnel, intervening every 2.5 miles [ 3 ] . This highlights the significant role of teleoperation in scaling AV fleets and the need for further research in this field. Figure 1 : An automated vehicle stops in front of an obstacle blocking the lane, unable to proceed without human intervention. In the presented approach, the vehicle automatically senses its need for teleoperation support through the usage of an arbitration graph for decision-making. A human remote operator provides guidance by modifying the planning constraints (red and green lines), enabling the vehicle to pass the obstacle. This scenario and results are presented in-depth in Section IV . Figure 2 : The arbitration graph used in this paper during active teleoperation. It is a simplified version of the arbitration graph introduced in [ 4 ] extended by the Teleoperation behavior component. Highlighted in green are currently active nodes, grayed out nodes are currently not applicable. Recent approaches to teleoperation focus on remote assistance, where ROs support specific ADS modules such as perception or planning through high-level instructions instead of controlling the vehicle directly via steering wheel and pedals. This reduces RO workload through event-based rather than continuous actions and increases safety [ 5 ] . However, these methods are deeply integrated into specific ADS architectures, limiting flexibility and extendability. To overcome these challenges, we propose a solution that seamlessly integrates human decision-making on planning level using arbitration graphs [ 6 ] . We implement teleoperation as an additional behavior component in the ADS stack that enables interaction with a planner without modifying existing planning components. As shown in Fig. 1 , when the AV encounters unsolvable scenarios, the teleoperation component engages. The RO can now modify planning constraints to allow the AV to pass the obstacle under human supervision after which the RO hands back control to the AV . Contributions Teleoperation Concept We extend the operational design domain (ODD) of an AV at runtime using human input on the planning level. Seamless Integration We utilize arbitration graphs to ensure compatibility with an existing ADS stack. HMI Concept We enable ROs to interact with the AV ’s environment model and planner constraints. Proof-of-Concept We demonstrate two scenarios solvable only with RO assistance in simulation. 3 II Related Work In an ADS , seamless integration of decision-making and trajectory planning layers is essential. These two components serve distinct yet interconnected roles. The decision-making layer determines the strategic behavior of the system, selecting maneuvers such as lane changes or speed adjustments based on the current situation. In contrast, the trajectory planning layer realizes these decisions by generating specific trajectories that satisfy both the behavioral goals and the physical constraints of the system. If no reasonable trajectory can be found by the ADS in a disengagement scenario, a human RO can support on planning level which will also be introduced in this chapter. II-A Decision-Making Historically, finite state machines have been a popular choice for decision-making due to their simplicity and ease of implementation [ 7 , 8 ] . However, their scalability is limited as the complexity of the system increases, leading to a massive growth in states and transitions. In contrast, behavior trees provide a scalable and transparent structure for behavior generation [ 9 ] . They allow for modular design and facilitate the addition of new behaviors without extensive reconfiguration. Despite these advantages, behavior tree lack inherent safety guarantees as their safety and robustness is heavily dependent on the arrangement of the nodes. Additionally, the return type of a behavior node is a simple binary or ternary value, which limits the flexibility of the system. Arbitration graphs are a hierarchical and modular decision-making framework designed for autonomous systems originally developed for robot soccer [ 10 ] . They have since been successfully applied to autonomous driving [ 6 , 4 ] . By combining simple atomic behavior components, they generate complex behavior in a bottom-up approach. This enables incremental integration of behavior components without modifying existing ones. Fig. 2 shows a visualization of an arbitration graph. Behavior components are the leaf nodes of the arbitration graph and compute a command given the current situation. Their invocation condition specifies whether a behavior component is applicable in the current situation provided by sensor data or an interpreted environment model. An activated behavior component can signalize that its execution can be continued using the commitment condition. The internal nodes, called arbitrators, decide between applicable options depending on the arbitrator’s decision policy. Arbitrators can be nested, allowing for a hierarchical decision-making structure. The icon in front of the arbitrator’s name in Fig. 2 indicates the decision policy: Automated Driving is a priority arbitrator whereas Urban Driving makes a decision based on the cost of its options. Arbitrators and behavior components share a generic interface which supports the integration of complex behavior commands such as trajectories. As a meta-framework, arbitration graphs offer great flexibility by allowing the integration of diverse underlying planning algorithms into a single decision-making structure. The modular structure of arbitration graphs allows for great maintainability and extensibility while ensuring a high level of transparency and interpretability. Safety can be reinforced through a tightly integrated verification step, which ensures that only valid and safe commands are executed [ 4 , 11 ] . This combination of flexibility, safety, and robustness makes arbitration graphs ideal for complex and dynamic applications such as autonomous driving

Read the complete text