2024 NASA SUITS Report: LLM-Driven Immersive Augmented Reality User Interface for Robotics and Space Exploration
Authors, Project Contributors: Kathy Zhuang, Zixun Huang, Yukun Song, Project Contributers: Yue Fan, Kai Mohl, Isaac Gonzalez, Yani Shi, Xiaowen Yuan, Rojan Kashani, Yunting Zhao, Aarav Desai, Ivan Apodaca, Advika Govindarajan, Ishaan Patel, Colin Strout, Caleb Strout, Emily Guo, Aeriel Amparo, Patr · 2025
Read 2024 NASA SUITS Report: LLM-Driven Immersive Augmented Reality User Interface for Robotics and Space Exploration by Authors, Project Contributors: Kathy Zhuang, Zixun Huang, Yukun Song, Project Contributers: Yue Fan, Kai Mohl, Isaac Gonzalez, Yani Shi, Xiaowen Yuan, Rojan Kashani, Yunting Zhao, Aarav Desai, Ivan Apodaca, Advika Govindarajan, Ishaan Patel, Colin Strout, Caleb Strout, Emily Guo, Aeriel Amparo, Patr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01206.txt.
Introduction
Kathy Zhuang and co-authors present an augmented reality system for space robots in 2025. The catalog creator field is garbled, but the head byline names six authors. Their system, built for a NASA student challenge, targets missions like Artemis.
A headset shows the interface while voice commands run through a large language model, keeping hands free. Tracking software fixes the robot's pose in space using a stereo camera dataset built for bending robot bodies. A mission console visualizes status for the ground team. A transformer model estimates full pose from depth data in real time, even under noise and blocked views.
Catalog record
| CREATOR | Authors, Project Contributors: Kathy Zhuang, Zixun Huang, Yukun Song, Project Contributers: Yue Fan, Kai Mohl, Isaac Gonzalez, Yani Shi, Xiaowen Yuan, Rojan Kashani, Yunting Zhao, Aarav Desai, Ivan Apodaca, Advika Govindarajan, Ishaan Patel, Colin Strout, Caleb Strout, Emily Guo, Aeriel Amparo, Patr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01206
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
2024 NASA SUITS Report: LLM-Driven Immersive Augmented Reality User Interface for Robotics and Space Exploration Authors, Project Contributors: Kathy Zhuang, Zixun Huang, Yukun Song, Project Contributers: Yue Fan, Kai Mohl, Isaac Gonzalez, Yani Shi, Xiaowen Yuan, Rojan Kashani, Yunting Zhao, Aarav Desai, Ivan Apodaca, Advika Govindarajan, Ishaan Patel, Colin Strout, Caleb Strout, Emily Guo, Aeriel Amparo, Patrick Lee, Yaamini Jois, Patrick Lasiter, Supervisors: Allen Y. Yang, Sheri L. Dragoo Abstract 1 University of California, Berkeley 2 Baylor University {kathy_zhuang, zixun, yukun_song, allenyang}@berkeley.edu 1 Project Overview As modern computers evolve to have better performance and larger computation power, new modes of interaction emerge along with novel designs of devices. Augmented Reality (AR), which allows users to interact with common objects via virtual interfaces, establishes a new space of designs and development and hence poses a significant challenge to machines’ spatial perception of the complex real-world environment. Pose estimation of 3D objects, as one of the most commonplace and important tasks in 3D perception, is still a challenging task for modern algorithms, especially in complex real environments with alternating scene properties and irregular object geometries. Our project is at the forefront of addressing critical challenges in human-robot interaction within dynamic mobile AR environments. We focus on exploring the potential ways of interacting with robots in space, especially in a non-intrusive manner. This interaction is made possible by integrating three key components: a non-intrusive head-mounted device serving as a user interface, voice control to enable astronauts to manipulate the interface and interact with robots using verbal commands, and a robot tracking algorithm that accurately localizes the robot’s position in 3D space. Enabled by these technologies, we proposed URSA, an LLM-driven immersive AR user interface for robotics and space exploration, and participated in the 2023 NASA Spacesuit User Interface Technologies for Students (NASA SUITS) [ 1 ] . This project aims to develop solutions for future spaceflight needs, particularly for the Artemis missions, which seek to establish a sustained human presence on the Moon and Mars. The NASA SUITS project focuses on developing user interfaces for spacesuits and mission control consoles, which are critical for the effective operation of rovers and other equipment on the Martian surface. Figure 1 : Structure of Augmented Reality Control Flow Industries that rely on robotics to perform complex tasks are the main stakeholders. Researchers in the fields of computer vision, robotics, and augmented/virtual reality have also invested heavily in similar challenge for object tracking in AR dynamic space. In a human-robot interaction setting, especially when dangerous or complex tasks are involved, there has been needs for intuitive interactions and precision. To foster intuitive interaction, we have explored multiple aspects of the user interfaces within AR devices. Display technologies, serving as the visual output for these devices, have seen extensive development, evidenced by pioneering products such as the Microsoft HoloLens [ 2 ] , Google Glass [ 8 ] , and HTC Vive XR Elite [ 3 ] . These technologies offer a spectrum of possibilities for presenting information to users. On the input side, modalities like gaze tracking, hand tracking, voice control, and physical controllers have been rigorously investigated for their efficacy in interfacing with AR devices. This paper focuses on non-intrusive AR interfaces by integrating the discreet visual output capabilities of the HoloLens [ 2 ] with the sophisticated interaction offered by large language models (LLMs) for voice control. To achieve precision, we used digital twin localization technology. For example, in the early versions of the Digital Twin Tracking Dataset (DTTD) [ 6 ] , and DTTD2 [ 13 ] , have set the groundwork for data collection under varied conditions and introduced algorithms that improve accuracy amidst real-world data noise. Similarly, by utilizing digital twin technology, we control robots via AR headset to perform accurate and dangerous tasks. Through such a control, the robot is capable of moving around to carry out work, while its position and poses being captured by external sensors and reflected on the AR headset simultaneously, enabling real-time remote control. This is particularly useful when robots need to be tracked with high accuracy without ground truth sensors. Our ongoing work emphasizes addressing the challenges of noise variance and real-time tracking accuracy with specialized camera systems like the ZED2, positioning our research at the forefront of digital twin applications in robotics. This focus provides novel insights and solutions in a rapidly evolving field. The development of robust datasets and algorithms for digital twin technology not only advances technical capabilities but also revolutionizes industrial processes, with a focus on industries that utilize complex robotics. Our specific solutions for realizing these are as follows: 1) We implemented an intuitive AR user interface using non-intrusive display and LLM-based voice control. 2) We created a comprehensive robot-specific dataset using the ZED2 camera, specifically designed for non-rigid bodies. 3) We designed an local Mission Control Console (LMCC) user interface to provide intuitive and real-time monitor of the space exploration task. 4) We deploy the dataset to the transformer-based 6DoF pose estimator (DTTDNet) with depth-robust designs on modality fusion and training strategies to enable real-time, high-level tracking accuracy. 5) We apply our datasets, model tracking, and associated technologies within the industry to fulfill the requirements of astronauts’ missions, and developed a comprehensive AR user interface, as demonstrated in Figure 3 . 2 Non-Intrusive AR User Interface for Head-Mounted Displays In the demanding and dynamic environment of space operations, the user interface for augmented reality (AR) systems, especially for astronauts, must prioritize minimal cognitive load and high efficiency. Astronauts require a system that maintains their awareness and focus on their surroundings, rather than diverts attention to manual controls or complex navigation menus. Our solution harnesses voice control to manage the AR interface non-intrusively, enhancing the interaction without overwhelming the user. 2.1 Non-Intrusive Design In the context of space exploration, where environmental awareness and visual clarity are paramount, our AR interface is designed to be non-intrusive, prioritizing the astronaut’s unobstructed view of their surroundings (Figure 2 ). Traditional AR interfaces often employ methods like gestures, touch pads, or gaze tracking. While these are effective on Earth, in the unique conditions of space, they can complicate interactions by cluttering the visual field or requiring manual inputs that are impractical in a bulky spacesuit. Figure 2 : Non-intrusive design for HMD (Showing the Suits Panel, EVA Task Panel and Virtual Voice Assistant – Ursa 2.2 Integration with Head-Mounted Display (HMD) For our augmented reality user interface, we selected the Hololens 2 [ 2 ] as the preferred head-mounted display (HMD) due to its specific features that cater to the safety and operational requirements of astronauts. The Hololens 2 [ 2 ] stands out among various HMD options for several critical reasons, particularly its non-intrusive design and fail-safe visibility features, which are essential in the challenging environment of space. One of the paramount concerns in designing AR interfaces for space is ensuring that astronauts maintain an unobstructed view of their surroundings. The Hololens 2 [ 2 ] is designed with a transparent visor, allowing users to see their environment directly, with digital overlays enhancing but not obscuring their real-world view. This transparency is crucial in space, where situational awareness is vital for safety and