Self-Supervised Monocular Visual Drone Model Identification through Improved Occlusion Handling
Stavrow A. Bahnam, Christophe De Wagter, Guido C.H.E. de Croon · 2025
Read Self-Supervised Monocular Visual Drone Model Identification through Improved Occlusion Handling by Stavrow A. Bahnam, Christophe De Wagter, Guido C.H.E. de Croon (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21695.txt.
Introduction
Stavrow Bahnam, Christophe De Wagter, and Guido de Croon teach drones to learn their own flight model from video, in a 2025 paper. Drones flying without GPS lean on cameras, but speed and nearby objects blur the view and block it. Filters that fill the gaps need an accurate model of the drone, usually learned with costly motion-capture rigs.
Their scheme learns from onboard video and flight data alone. A self-taught pose estimator first acts as teacher for the flight model. A new method for handling blocked views sharpens the teacher, cutting odometry error by about fifteen percent on average. The student model then supports flight in fast, cluttered settings. Removing the motion-capture step could let each drone calibrate itself wherever it flies.
Catalog record
| CREATOR | Stavrow A. Bahnam, Christophe De Wagter, Guido C.H.E. de Croon |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21695
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Supervised Monocular Visual Drone Model Identification through Improved Occlusion Handling Stavrow A. Bahnam, Christophe De Wagter, Guido C.H.E. de Croon Abstract Ego-motion estimation is vital for drones when flying in GPS-denied environments. Vision-based methods struggle when flight speed increases and close-by objects lead to difficult visual conditions with considerable motion blur and large occlusions. To tackle this, vision is typically complemented by state estimation filters that combine a drone model with inertial measurements. However, these drone models are currently learned in a supervised manner with ground-truth data from external motion capture systems, limiting scalability to different environments and drones. In this work, we propose a self-supervised learning scheme to train a neural-network-based drone model using only onboard monocular video and flight controller data (IMU and motor feedback). We achieve this by first training a self-supervised relative pose estimation model, which then serves as a teacher for the drone model. To allow this to work at high speed close to obstacles, we propose an improved occlusion handling method for training self-supervised pose estimation models. Due to this method, the root mean squared error of resulting odometry estimates is reduced by an average of 15 % 15\% . Moreover, the student neural drone model can be successfully obtained from the onboard data. It even becomes more accurate at higher speeds compared to its teacher, the self-supervised vision-based model. We demonstrate the value of the neural drone model by integrating it into a traditional filter-based VIO system (ROVIO), resulting in superior odometry accuracy on aggressive 3D racing trajectories near obstacles. Self-supervised learning of ego-motion estimation represents a significant step toward bridging the gap between flying in controlled, expensive lab environments and real-world drone applications. The fusion of vision and drone models will enable higher-speed flight and improve state estimation, on any drone in any environment. I introduction Accurate ego-motion estimation is crucial for controlling autonomous drones. However, this is still challenging in GPS-denied areas. An often employed approach is to have the drone perform Visual Inertial Odometry (VIO), which makes use of a camera and an inertial measurement unit (IMU) to determine its velocity and rotation rates [ 1 ] . Since VIO only requires lightweight sensors (a camera and IMU), it is suitable for drones that have Size Weight and Power (SWaP) restrictions, like Micro Air Vehicles (MAVs). Fig. 1 : We introduce a self-supervised learning approach that only uses data from onboard the drone to learn ego-motion estimation. Specifically, this data consists of the inertial measurements and control commands available in the flight controller (left), and monocular camera images (right). The approach does not require any external infrastructure and allows to scale the monocular vision-based estimates. For certain drone applications, the drone requires a high flight speed. For example, for search and rescue, it would be desirable to find the victims as fast as possible and to cover the maximum amount of area with a single battery. Fast flight does make visual navigation more difficult, as it leads directly to a higher optical flow and motion blur in images. The effects of occlusions also increase when flying very fast near obstacles, as this increases the portion of pixels that cannot be matched from one frame to the next. This is problematic for traditional VIO methods that detect and track features across frames like ROVIO [ 2 ] . Learning-based VIO has the potential to be more robust to fast-flight conditions, leveraging the capability of deep neural networks to cope with degraded images, cf. [ 3 , 4 , 5 ] . However, learning-based VIOs are not as accurate as traditional VIO methods yet and are computationally more expensive, requiring an onboard computer with a GPU, like an NVidia Jetson Orin NX. Autonomous drone racing is the perfect scenario to test the performance of state estimation algorithms during fast motion. While detection of racing gates can be used to infer the drone’s absolute position on the track, gates may not be visible during substantial parts of the flight. Hence, odometry based on ego-motion estimation is important for continuous position feedback control. In the AlphaPilot 2019 race, the runner-up solution employed a filter-based visual-inertial odometry method based on ROVIO [ 2 ] . However, the authors noted in [ 6 ] that at higher speeds VIO tracking failures resulted in crashes. The winning solution used model -based inertial odometry, combining a drone model with IMU measurements [ 7 ] . This type of model is not as accurate as VIO, but was shown to increase the long-term stability of the motion-prediction. Such a drone model is typically estimated based on data with available ground-truth positions and velocities, as obtained using a motion capture system. For example, based on motion tracking data, in [ 8 ] a quadrotor model was learned and fused with an inertial odometry model to get accurate state estimation. Besides the need for an external motion tracking system, the use of global coordinates also limits the approach in [ 8 ] to known trajectories. This limitation also applies to Swift [ 9 ] , the seminal autonomous drone system that for the first time beat human FPV (First Person View) drone racing champions. The state estimation of Swift consists of two parts: VIO from an Intel RealSense T265 [ 10 ] and gate detection to correct for the drift. Furthermore, the authors model a Gaussian noise of the VIO, by fitting it through the position residual obtained from external motion capture measurements to refine the training of their control policy. Visual odometry can behave differently in different environments, due to changes in illumination and the amount of texture. Therefore, for a new environment the control of Swift would need to be retrained with a newly estimated Gaussian noise to keep up the same performance. However, motion capture is not available everywhere. In another recent study, reinforcement learning was used to directly map racing-gate-segmentation images to control commands for drone racing [ 11 ] . The trained neural network successfully exploited the specific properties of the segmented racing gates to seamlessly transfer from simulation to an identical track in the real world. The generalization of this method to other applications will be limited, though. In contrast, humans can learn to estimate the state of a drone from only monocular FPV video stream combined with own control inputs. After getting used to a particular drone, this experience seamlessly transfers to new environments, including a-priori unknown environments. Motivated by this, in this article, we train a self-supervised visual ego estimation network for autonomous drone racing using a monocular video only, such that it can learn in any environment. To also make this possible in racing environments at very high speeds close to obstacles, we first improve existing self-supervised learning (SSL) methods for ’pose’ estimation (the translation and rotation between two images). In particular, we propose a novel approach for dealing with parts of the environment that are occluded and with pixels moving out of view - phenomena that are exacerbated by agile drone flight. Moreover, we investigate the self-supervised learning of a neural drone model that uses inertial measurements and motor feedback as inputs. We show that the learned drone model can predict