SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum
JunEn Low, Maximilian Adang, Javier Yu, Keiko Nagami, Mac Schwager · 2024
Read SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum by JunEn Low, Maximilian Adang, Javier Yu, Keiko Nagami, Mac Schwager (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16346.txt.
Introduction
JunEn Low and co-authors present a full system for teaching drones to fly by sight alone, in a 2024 paper. Their package, called SOUS VIDE, joins a simulator, a training method, and a control design. Trained policies moved from simulation to real flights with no extra tuning. They run on the drone's own sensors and computer.
The simulator pairs a simple flight model with rich 3D scenes rebuilt from photos. It can render flights at up to 130 frames per second. An expert controller flew hundreds of thousands of test runs there. A small neural network then learned to copy that expert from camera, motion, and sensor data. It issues thrust commands twenty times per second in real flight.
Catalog record
| CREATOR | JunEn Low, Maximilian Adang, Javier Yu, Keiko Nagami, Mac Schwager |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-16346
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum JunEn Low, Maximilian Adang, Javier Yu, Keiko Nagami, Mac Schwager Abstract We propose a new simulator, training approach, and policy architecture, collectively called SOUS VIDE, for end-to-end visual drone navigation. Our trained policies exhibit zero-shot sim-to-real transfer with robust real-world performance using only onboard perception and computation. Our simulator, called FiGS, couples a computationally simple drone dynamics model with a high visual fidelity Gaussian Splatting scene reconstruction. FiGS can quickly simulate drone flights producing photorealistic images at up to 130 fps. We use FiGS to collect 100k-300k image/state-action pairs from an expert MPC with privileged state and dynamics information, randomized over dynamics parameters and spatial disturbances. We then distill this expert MPC into an end-to-end visuomotor policy with a lightweight neural architecture, called SV-Net. SV-Net processes color image, optical flow and IMU data streams into low-level thrust and body rate commands at 20 Hz onboard a drone. Crucially, SV-Net includes a learned module for low-level control that adapts at runtime to variations in drone dynamics. In a campaign of 105 hardware experiments, we show SOUS VIDE policies to be robust to 30% mass variations, 40 m/s wind gusts, 60% changes in ambient brightness, shifting or removing objects from the scene, and people moving aggressively through the drone’s visual field. Code, data, and experiment videos can be found on our project page: https://stanfordmsl.github.io/SousVide/ . I INTRODUCTION Learned visuomotor policies offer a compelling alternative to traditional drone navigation stacks by unifying perception and control into a streamlined framework. Unfortunately, training policies with human-like agility and collision avoidance requires a large corpus of visual and state data, making behavior cloning from human pilot demonstrations impractical. Simulation provides a promising alternative, but the sim-to-real gap has remained a persistent obstacle to real-world deployment. Recent work has demonstrated that in controlled environments and with carefully built digital twins in simulation, learned policies can achieve highly agile, superhuman performance [ 1 , 2 , 3 ] . However, this raises the question: can we train a visuomotor policy to navigate unstructured real-world environments with minimal human curation? Fig. 1: SOUS VIDE overview: We train our FiGS simulator from a hand held camera. We use FiGS to generate flight demonstrations (image/state-action pairs) from an MPC expert with privileged information randomized over dynamics parameters and positional disturbances. We use this data to train our policy, SV-Net, which operates solely with onboard observations. We address this challenge with FiGS (Flying in Gaussian Splats), a photorealistic drone simulator combining a Gaussian Splat (GSplat) [ 4 ] scene model with a lightweight 10-dimensional drone dynamics model using thrust and body rate inputs—equivalent to the Acro mode used by expert human pilots. FiGS reconstructs scenes from video captures, like publicly available footage or smartphone recordings, processed with standard tools [ 5 ] and generates realistic image sequences and state estimation data for a drone at up to 130 fps on a standard GPU, all within an hour of acquiring video data. This contrasts with the current practice of approximating each real scene with a handcrafted simulation instance [ 6 , 7 , 8 , 9 , 10 , 11 ] that can take weeks of laborious sim-to-real transfer to perfect. Building on FiGS, we develop SOUS VIDE 2 2 2 The name is inspired by the sous vide cooking technique that evenly cooks food in a vacuum-sealed bag under carefully controlled conditions. (Scene Optimized Understanding via Synthesized Visual Inertial Data from Experts), a behavior cloning pipeline that produces a robust drone navigation policy capable of zero-shot sim-to-real transfer—trained entirely within simulation without real-world demonstrations or fine-tuning (Fig. 1 ). Specifically, we use FiGS to generate 100k-300k image/state-action pairs from an expert MPC policy following a desired nominal trajectory within a GSplat. The MPC has access to the ground truth state in the simulator and is therefore able to demonstrate high-quality, collision-free trajectories. To obtain stable and robust flight, we randomize dynamics parameters and spatial perturbations and record the MPC’s response. We use these expert demonstrations to train a student policy without privileged information (no lateral position data). The learned policy produced by SOUS VIDE, SV-Net, is a novel architecture designed to process both images and observable state history while remaining efficient enough to run onboard the drone. The policy ingests images using a SqueezeNet [ 12 ] , re-trained on our data, and outputs a feature vector that is fused with observable data across several small Multi-Layer Perceptrons (MLPs) to produce low-level body-rate commands. Within these MLPs, we also implement a form of the Rapid Motor Adaptation (RMA) concept from [ 13 ] , which takes a history of observable data from a sliding window and produces a latent code that captures evolving flight dynamics in real time. We find this RMA module is crucial for robust flight, adapting online to variations such as battery drain, rotor downwash effects, and wind gusts. To summarize, we make the following contributions: 1. Flying in Gaussian Splats (FiGS) : A simulator coupling GSplat scene models with drone dynamics for efficient and photorealistic visual-inertial data generation. 2. Scalable Visuomotor Policy Generation : We use FiGS to generate large synthetic datasets to train visuomotor policies that transfer zero-shot to real-world flight. 3. SV-Net : An onboard policy that fuses image and observable states to infer thrust and body rates while continuously adapting to changing flight conditions. We evaluate SOUS VIDE policies across 105 hardware flights in 4 different scenes, testing 9 different experimental conditions. We demonstrate our policy’s robustness to 30% mass variations, 40 m/s wind gusts, 60% changes in ambient brightness, shifting or removing objects from the scene, and people moving aggressively through the drone’s visual field. The paper is organized as follows: Section II reviews related work, Section III describes the FiGS simulator, Section IV details the MPC-based data synthesis, and Section V presents the SV-Net policy. Hardware experiments are in Section VI , with conclusions, limitations, and future work in Section VII . II Related Work Drone Polices Trained with GSplats: Learned representations like GSplats have proven effective in training visuomotor policies across many domains, from manipulation [ 14 , 15 ] to bipedal locomotion [ 16 ] to aerial robotics [ 17 , 18 ] . Closest to our approach, [ 17 ] uses a learned representation and an MPC expert to train a trajectory-following policy but requires an initial sample of real-world expert flight demonstrations to be collected via motion capture. Additionally, its 45^{\circ} downward-facing camera focuses on ground features, missing the spatial information of a forward view needed for obstacle avoidance, as seen in drone racing [ 2 ] . Meanwhile, [ 18 ] , treats the GSplat reconstruction as a background, relying instead on colored spheres as visual markers injected into both the simulation and real-world scenes as the basis for decision making. Moreover, it relies on velocity commands that encode only high-level approach and turn behaviors, delegating low-level control to a manufacturer-supplied autonomy stack. To the best of our knowledge, SOUS VIDE is the first method to leverage GSplats for generating low-level drone navigation policies for unstructured environments without assistive infrastructure or real-world expert flight data. Training