VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots

Kushagra Srivastava, Rutwik Kulkarni, Manoj Velmurugan, Nitin J. Sanket · 2025

Read VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots by Kushagra Srivastava, Rutwik Kulkarni, Manoj Velmurugan, Nitin J. Sanket (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22876.txt.

Introduction

Kushagra Srivastava and co-authors build a flight simulator for drone students that cannot crash, in a 2025 paper. Hands-on courses in aerial robotics need testbeds where learners can experiment without smashing hardware. Their framework, VizFlyt, feeds a real drone's motion, tracked by an external positioning system, into a photorealistic virtual world built with 3D Gaussian splatting. The drone flies physically in a safe space while seeing a rich simulated scene.

The loop updates faster than one hundred times per second, quick enough for perception algorithms to react naturally. The team draws on their own teaching experience to propose an open-source, open-hardware curriculum built around the system. Trials in real course projects showed students could stress-test autonomy code stress-free. The paper packages the code, datasets, and build guides for other instructors.

Catalog record

CREATORKushagra Srivastava, Rutwik Kulkarni, Manoj Velmurugan, Nitin J. Sanket
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22876

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VizFlyt: Perception-centric Pedagogical Framework For Autonomous Aerial Robots Kushagra Srivastava, Rutwik Kulkarni, Manoj Velmurugan, Nitin J. Sanket Abstract Autonomous aerial robots are becoming commonplace in our lives. Hands-on aerial robotics courses are pivotal in training the next-generation workforce to meet the growing market demands. Such an efficient and compelling course depends on a reliable testbed. In this paper, we present VizFlyt , an open-source perception-centric Hardware-In-The-Loop (HITL) photorealistic testing framework for aerial robotics courses. We utilize pose from an external localization system to hallucinate real-time and photorealistic visual sensors using 3D Gaussian Splatting. This enables stress-free testing of autonomy algorithms on aerial robots without the risk of crashing into obstacles. We achieve over 100Hz of system update rate. Lastly, we build upon our past experiences of offering hands-on aerial robotics courses and propose a new open-source and open-hardware curriculum based on VizFlyt for the future. We test our framework on various course projects in real-world HITL experiments and present the results showing the efficacy of such a system and its large potential use cases. Code, datasets, hardware guides and demo videos are available at https://pear.wpi.edu/research/vizflyt.html I Introduction The use of aerial robots (drones) has surged in the past decade, with applications ranging from search and rescue [ 1 ] to farming [ 2 ] and inspection [ 3 ] . However, most drones are manually operated, limiting their scalability and use in harsh conditions. Hence, there is a growing need for autonomous drones and the software to operate them. Aerial robotics courses [ 4 , 5 , 6 , 7 , 8 , 9 , 10 ] play a pivotal role in training students from K-12 to graduate levels, focusing on skills from piloting to autonomy software development. While many courses teach manual piloting [ 11 , 12 ] or drone hardware building [ 4 ] , fewer focus on real-time autonomy using onboard sensing and computation. Hands-on learning has consistently proven effective for robotics education, and aerial robotics is no exception. However, running a hands-on drone autonomy course faces challenges such as (a) procuring the right hardware platform with long-term support, (b) ensuring safe operation for students, (c) minimizing costs for operation, maintenance, and repair, (d) providing an extensive and user-friendly API for sensor and control access, (e) allowing modularity for adding new sensor/computational payloads, and (f) ensuring stability, robustness, and repeatability. Depending on the course’s focus, some factors may outweigh others. For example, in courses focused on teaching controls and planning, it might be possible to avoid using onboard sensors and simply use an external localization system mated to an offboard PC that communicates to the robot via a wireless link. Furthermore, in courses that involve sensor fusion or perception, it becomes essential to use onboard sensors/computation due to the requirement of high rate and low latency processing to hit the control targets required. This paper focuses on developing a scalable, robust framework for teaching vision-based drone autonomy, which is widely used for its cost-effectiveness and ubiquity. Currently, aerial robotics courses rely on custom-built or educational drone platforms [ 13 , 14 , 15 ] , many of which are discontinued. This presents significant logistical challenges, as course materials must be regularly updated to accommodate new hardware. The issue is compounded by hardware damage from collisions with environmental elements, leading to costly spare part replacements. Additionally, an autonomy-focused aerial robotics course requires integrating hardware like front-facing, downward-facing, and/or stereo cameras. This added complexity increases both the effort and cost of running the course, making it difficult to scale for larger student groups. A solution is to conduct the course entirely on a simulator [ 16 , 17 , 18 ] . We encourage reading [ 19 ] for a study on aerial robot simulators. While simulators can effectively replicate certain aspects of drone operation, they fall short in providing the real-world experience needed to develop robust, reliable algorithms for safety-critical systems in aerial vehicles. Recent research [ 20 , 21 , 22 ] has utilized Hardware-In-The-Loop (HITL) simulation, where a physical robot flies in the real world, eliminating the need to simulate flight dynamics. This approach is more flexible than using real robots with onboard sensing and computation, as it allows for quick adaptation to various environmental settings, reduces collision risk, and retains flight dynamics fidelity. Inspired by this, we combine a photorealistic rendering framework with HITL to create a scalable method for teaching. This enables students to focus on autonomy concepts at any level (perception, planning, and controls) in a realistic environment. For instructors and teaching assistants, it reduces the logistical burden (e.g., maintaining drones and experimental setups), lowers operational costs, minimizes repair times, and simplifies live-grading efforts. The authors have collectively participated in five aerial robotics courses [ 10 , 8 , 9 ] across three universities (University of Pennsylvania, University of Maryland, College Park, and Worcester Polytechnic Institute) in roles ranging from student to teaching assistant to instructor. These courses spanned multiple departments (Robotics Engineering, Computer Science, Aerospace Engineering, and Mechanical Engineering) and catered to both undergraduate and graduate students. Our experiences have provided us with a well-rounded perspective on running a hands-on drone autonomy course with minimal logistical challenges. We propose VizFlyt , a perception-centric pedagogical framework for autonomous aerial robotics based on HITL concepts. By leveraging 3D Gaussian Splatting (3DGS) [ 23 ] for real-time view synthesis from robot pose, VizFlyt decouples aerial robotics coursework from hardware dependency, providing a flexible, robust, and cost-effective solution for testing in complex visual environments. It is an intuitive framework designed for ease of use, expandable to novel features, requiring no prior background knowledge, making it a perfect framework for teaching aerial robotics. The key contributions of the VizFlyt framework are: • VizFlyt eliminates the need for onboard proprioceptive sensors, reducing hardware complexity and reliance on specific robots, while facilitating RGBD updates at over 100 Hz per sensor. • VizFlyt enables rapid dynamic asset generation for novel environment creation using synthetic or real data, supporting flight testing across a variety of scenarios. • We demonstrate VizFlyt ’s effectiveness for teaching aerial robotics through key tasks such as Visual Odometry (VO), high-speed obstacle avoidance, and navigation through known and unknown gaps in real-world experiments. • The VizFlyt framework (software and hardware) will be open-source, making it future-proof and expandable. II Learnings From Current Curricula II-A Others’ Aerial Robotics Coursework Several aerial robotics courses offer first-hand experience and often involve 3 distinctive modules (i) Development and assembly of drone hardware, (ii) Flying and operating drones, (iii) Development of drone autonomy software. Drone autonomy encompasses tasks or objectives focused on motion planning [ 10 , 6 , 8 ] , control systems [ 10 , 4 , 7 , 8 ] , and perception algorithms [ 10 , 5 , 4 , 7 , 8 ] . Course projects are tailored around fundamental tasks such as SLAM, sensor fusion, quadrotor dynamics, path planning, pose estimation, and obstacle avoidance. While some of the projects provide an introductory understanding of quadrotor operation, others are more advanced, catering to graduate-level expertise. Some of the courses focus on software integration for robust performance. However, testing under various conditions of these projects is limited due

Read the complete text