A U R A : Multimodal Shared Autonomy for Real-World Urban Navigation

Yukai Ma, Honglin He, Selina Song, Wayne Wu, Bolei Zhou · 2026

Read A U R A : Multimodal Shared Autonomy for Real-World Urban Navigation by Yukai Ma, Honglin He, Selina Song, Wayne Wu, Bolei Zhou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-01659.txt.

Introduction

Ma and colleagues present AURA, a multimodal shared autonomy method for urban navigation that splits the job into human-led direction and AI-led control. Existing shared autonomy approaches usually put humans and AI in one action space, which raises cognitive load. AURA's Spatial-Aware Instruction Encoder lines up varied human instructions with what the system sees and where things are. Tests in simulation and the real world show AURA follows human instructions, reduces manual effort, and improves navigation stability. Shared autonomy reduces takeovers by more than 44 percent.

Catalog record

CREATORYukai Ma, Honglin He, Selina Song, Wayne Wu, Bolei Zhou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53052
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.01659

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A U R A : Multimodal Shared Autonomy for Real-World Urban Navigation Yukai Ma, Honglin He, Selina Song, Wayne Wu, Bolei Zhou Abstract Long-horizon navigation in complex urban environments relies heavily on continuous human operation, which leads to fatigue, reduced efficiency, and safety concerns. Shared autonomy, where a Vision-Language AI agent and a human operator collaborate on maneuvering the mobile machine, presents a promising solution to address these issues. However, existing shared autonomy methods often require humans and AI to operate within the same action space, leading to high cognitive overhead. We present Assistive Urban Robot Autonomy (AURA), a new multi-modal framework that decomposes urban navigation into high-level human instruction and low-level AI control. AURA incorporates a Spatial-Aware Instruction Encoder to align various human instructions with visual and spatial context. To facilitate training, we construct MM-CoS , a large-scale dataset comprising teleoperation and vision-language descriptions. Experiments in simulation and the real world demonstrate that AURA effectively follows human instructions, reduces manual operation effort, and improves navigation stability, while enabling online adaptation. Moreover, under similar takeover conditions, our shared autonomy framework reduces the frequency of takeovers by more than 44%. Demo video and more detail are provided in the project page. 1 Introduction Figure 1 : Shared Autonomy for Urban Navigation. We introduce AURA, a dual-system VLA for shared autonomy in urban navigation. AURA not only follows instructions but also enables human users to guide and correct a robot in real time through various visual and language instructions. Despite rapid progress in autonomous driving on roads, AI remains significantly challenging to safely operate mobile machines in public urban spaces, such as sidewalks, parks, and school campuses, due to the complexity of the surroundings and the diverse range of human activities. Thus, many existing sidewalk mobile machines (e.g., assistive wheelchairs and food delivery bots) still rely heavily on a human-in-the-loop approach. For example, remote workers may teleoperate or closely supervise delivery bots to accomplish last-mile tasks. In these settings, the human pilot must remain fully attentive to both control and situational awareness. Road hazards, such as rugged terrain and fragmented curbs, as well as unexpected pedestrian behavior and other human-made errors, pose significant risks to mobile machines and their human pilots. To address these challenges, recent studies have explored the paradigm of shared autonomy, where AI agents assist and augment human pilots in maneuvering machines in both training 11 ; 12 ; 46 and test-time phases 60 . For long-horizon navigation tasks, researchers are increasingly seeking AI assistant models that can effectively reduce human workload, allowing humans to remain in the loop primarily for monitoring and failure recovery 55 ; 35 ; 51 ; 22 . Deploying navigation autonomy models in the real world for long-horizon navigation typically involves switching between full human operation and full AI control 11 ; 12 ; 46 ; 22 ; 55 ; 35 . However, these approaches assume that the human and AI operate in the same low-level action space that directly controls the driving wheels and pedals, thereby requiring the human to operate at the same frequency as the AI. This coupling becomes inefficient and cognitively demanding for long-horizon tasks such as urban sidewalk food delivery. We believe a more effective paradigm is a division of labor between humans and AI by decomposing urban navigation into hierarchical levels: (i) humans provide high-level instructions, such as reasoning about corner cases and proposing alternative routes, and (ii) AI agents make low-level executions, such as lane keeping and obstacle avoidance. By delegating low-level control to the AI while reserving high-level strategic decisions for the human, this human-AI shared autonomy framework will achieve improved efficiency, safety, and robustness in long-horizon urban navigation. In this work, we propose A ssistive U rban R obot A utonomy ( AURA ), a multi-modal shared autonomy framework designed to understand human instructions and handle low-level control, thereby significantly reducing human operation costs. Specifically, AURA is a Vision-Language-Action (VLA) model with a dual-system architecture. Acting as an “autonomous assistant,” AURA can be seamlessly integrated into existing delivery robots without any hardware modifications. For high-level instruction, the system enables various human instructions through a vision-and-language interface, where concise natural-language or visual prompts on live video streams replace laborious joystick control. It enables safe, scalable sidewalk autonomy while substantially reducing operator workload. As illustrated in the center of Fig. 1 , AURA can interpret multiple forms of human instructions: (i) texting, where users describe the intent; (ii) drafting, where users draw a rough path on the observation view; and (iii) arrowing, where users demonstrate the desired speed and direction. A key challenge in shared autonomy is interpreting ambiguous human instructions and grounding them within the surrounding spatial context. To address this, we introduce a Spatial-Aware Instruction Encoder (SIE) that explicitly aligns textual instructions with both the semantic layout and the geometric structure of the scene. This design addresses the spatial-understanding limitations of standard vision–language models (VLMs), enabling the model to reason about the user’s intent and where those instructions can be executed, thereby improving robustness across diverse real-world environments. To support this model training, we construct a multimodal video dataset, MM-CoS, on top of our sidewalk teleoperation dataset CoS 19 that includes a total of 50 hours of high-quality teleoperation data collected on real-world sidewalks. The dataset spans a diverse range of scenarios, including different cities, weather conditions, and lighting variations. We further curate relevant clips using behavior- and ego-state-based filters to ensure that the resulting human control behaviors are representative. By combining human instructions inferred from VLMs with ground-truth trajectories from human operators, we obtain a unified dataset in which multimodal human instructions serve as inputs and expert trajectories serve as outputs. Experiments show that our framework accurately interprets and executes human instructions, achieving over 15% lower L2 error and reducing human operation costs by more than 70% compared to baselines. These results highlight the effectiveness of our shared-autonomy design in enabling efficient and human-aligned navigation. In summary, our contributions are threefold: • A multi-modal shared autonomy framework that integrates human instruction following with low-level control via a unified VLM encoder and diffusion policy. • We introduce Spatial-Aware Instruction Encoder designed for instruction understanding, which is trained on a new dataset with multi-modal instructions to connect human teleoperation and VLM-based intention inference. • Simulated and real-world experiments demonstrate the effectiveness of our approach in following instructions, improving stability, and reducing human operation cost. 2 Related Work Human-in-the-loop Learning and Shared Autonomy. It is crucial to ensure safety when deploying robots in urban environments, and incorporating human preferences into decision-making offers a promising way toward trustworthy, human-centered autonomy 46 ; 11 . Prior works have incorporated human feedback during training to improve policy alignment or to correct undesired behaviors, a paradigm often referred to as learning with human involvement or human-in-the-loop learning 67 ; 1 ; 62 ; 64 ; 46 ; 11 ; 45 . In this paradigm, humans actively intervene or provide demonstrations when the robot exhibits unsafe or suboptimal behaviors, allowing the policy to learn corrective actions and refine itself through shared autonomy. During deployment, it is equally important for the robot to understand implicit and explicit human intentions and preferences, expressed through

Read the complete text