FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking
Yan Miao, Karteek Gandiboyina, Noah Giles, Hideki Okamoto, Bardh Hoxha, Georgios Fainekos, Sayan Mitra · 2026
Read FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking by Yan Miao, Karteek Gandiboyina, Noah Giles, Hideki Okamoto, Bardh Hoxha, Georgios Fainekos, Sayan Mitra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29783.txt.
Introduction
Miao and six colleagues present FalconTrack, a framework for vision-based aerial tracking in GPS-denied environments. Reliable tracking perception needs large labelled datasets, yet photorealistic ones usually rely on heavy manual annotation. FalconTrack attacks both the data problem and the tracking problem together, aiming for zero-shot transfer from simulation to real hardware.
An automated labelling pipeline inside a Gaussian Splatting simulator isolates target objects from short videos and composites them with randomized backgrounds. It produces RGB images with mask, class, and pose labels, generating about ten thousand labelled images in under twenty minutes. A multi-head perception module trains on that data, and its outputs fuse with dynamics priors inside an extended Kalman filter. The perception model reaches near-perfect class accuracy under zero-shot sim-to-real transfer. On real hardware, closed-loop tracking succeeds in all test trajectories.
Catalog record
| CREATOR | Yan Miao, Karteek Gandiboyina, Noah Giles, Hideki Okamoto, Bardh Hoxha, Georgios Fainekos, Sayan Mitra |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.29783
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FalconTrack: Photorealistic Auto-Labeled Perception and Physics-Aware Vision-Based Aerial Tracking Yan Miao, Karteek Gandiboyina, Noah Giles, Hideki Okamoto, Bardh Hoxha, Georgios Fainekos, Sayan Mitra Abstract Vision-based aerial tracking is critical in GPS-denied environments. Reliable perception for tracking depends on large-scale labeled data, yet most photorealistic datasets rely on heavy manual annotation and are time-consuming to produce. We present FalconTrack , a unified perception-and-tracking framework that (i) leverages a photorealistic editable simulator for automated label generation and (ii) combines multi-head perception with physics-aware tracking for zero-shot sim-to-real transfer. FalconTrack provides an automated labeling pipeline in a Gaussian Splatting simulator that isolates target Gaussians from short object videos and composites them with randomized backgrounds to generate RGB, mask, class, and 6-DoF pose labels, producing about 10k labeled images in under 20 minutes. Using this dataset, we train a multi-head perception module with staged learning and reprojection consistency, and fuse its outputs with class-conditioned dynamics priors in an EKF for tracking. Our perception model outperforms two baselines and reaches 96–100% class accuracy in zero-shot sim-to-real transfer on three geometrically diverse objects and two environments, while maintaining consistent performance in unseen simulated and real scenes. In real hardware closed-loop visual tracking, the onboard system runs at about 25 Hz and achieves 100% success in sim-to-real F1-tenth and gate tracking in five trajectories across two environments, while a mask-centered vision baseline drops to 60% success on F1-tenth during fast out-of-view scenarios. I Introduction Autonomous vision-based aerial tracking is important for applications such as transportation, environmental monitoring, and search and rescue, especially in GPS-denied environments where ground-truth state is unavailable. Fig. 1 : FalconTrack for Real-World Tracking: We show a time-lapse of an airborne ego quadrotor tracking a ground F1-tenth [ 22 ] target vehicle during a left-turn trajectory using vision. Time labels (T0–T2) use black text for ego snapshots and red text for target snapshots; the left insets show onboard RGB images with predicted masks and pose estimates at T0–T2, and the top-right plot visualizes the 3D trajectories of the ego (black) and target (red). Fig. 2 : Auto-Labeling Pipeline for a Photorealistic Perception Dataset: From a 2-minute real video per object, we construct object GSplats and combine them with diverse background GSplats and domain randomization using the FalconGym 2.0 Edit API [ 19 ] . Using known camera matrices and rendering transforms, we automatically compute pixel-accurate masks and 6-DoF poses for each frame, yielding a large-scale, diverse training dataset of RGB images, masks, class labels, and 6-DoF poses without manual annotation. Reliable vision-based tracking typically relies on accurate perception models trained on large-scale labeled datasets. However, many existing perception datasets rely heavily on human annotation, which is costly, time-consuming, potentially error-prone, and difficult to scale to new irregular objects (non-analytic shapes or non-parametric geometry). For example, ImageNet [ 5 ] , widely used to train object detectors and classifiers such as YOLO [ 24 ] , required thousands of Amazon Mechanical Turk workers [ 32 ] to manually label categories; BDD100K [ 34 ] , widely used for self-driving policy training, also relies on human annotation for 100K images with bounding boxes and lane markings. Some simulators (e.g., CARLA [ 7 ] and Gazebo [ 13 ] ) can auto-generate labels. For example, KITTI-CARLA [ 6 ] automatically provides labels for lidar, instance segmentation, and 3D object bounding boxes in CARLA. However, images from such simulators often lack photorealism, causing a sim-to-real gap when trained perception is deployed in the real world. Diffusion-based generative models such as GLIDE [ 21 ] can synthesize photorealistic 2D images from text, but they do not guarantee that generated objects match the target appearance and more importantly do not provide 6-DoF pose labels. Recent advances in photorealistic simulators based on 3D Gaussian Splatting [ 12 ] , such as FalconGym 2.0 [ 19 ] , show promise by training a gate mask detector in simulation and deploying zero-shot in the real world. However, their gate labeling remains semi-manual, relying on measured gate size and analytic geometry (e.g., a ring defined in a canonical frame), and is demonstrated on a single gate object rather than irregular targets (e.g., an F1-tenth vehicle or a quadrotor). To bridge the gap between photorealistic image synthesis and automated labeling of irregular objects, we propose an automated labeling pipeline. From a 2-minute video, we reconstruct a 3D Gaussian Splatting scene, use SAGA [ 3 ] to isolate target-related splats, and fuse them with diverse background GSplats under domain randomization to improve generalization and reduce sim-to-real gaps. Given the camera intrinsics and extrinsics used for rendering, we project the 3D Gaussians into the image plane to generate pixel-accurate masks and compute the 6-DoF target pose directly from the known rendering transform. This yields a photorealistic and auto-labeled dataset for perception of irregular objects without any manual annotation. A well-labeled perception dataset can provide the basis for robust closed-loop vision-based aerial tracking across diverse target types and environments. Drone racing is a representative special case of tracking a sequence of static gates. Prior racing work [ 8 ] [ 11 ] trains a gate detector and connects it to downstream neural control; although it can reach high speed (e.g., 11 m/s), it does not generalize to new tracks or object types. FalconGym 2.0 [ 19 ] uses a single policy to traverse different tracks but still focuses on one object type (gate). A transformer-based unified perception module [ 26 ] generalizes tracking across target types and uses a model-free controller to follow the detected 2D bounding-box center, but it does not exploit target-specific dynamics and 6 DoF pose estimates. To address these limitations, we move beyond per-object 2D detection and design a perception module that predicts target class and full 6-DoF pose from our auto-labeled data, trained with staged learning and a reprojection-consistency loss. Building on this 6-DoF perception, we perform closed-loop tracking of three target objects in simulation and real experiments, using class-conditioned dynamics priors to smooth pose estimates and recover from temporary out-of-view events. Experiments show that our unified perception module classifies three irregular objects with 100% accuracy in simulation and reaches 96–100% class accuracy when deployed zero-shot to the real world. Qualitative results in Figure 3 also show consistent perception in unseen simulated and real environments. In vision-based closed-loop tracking, our physics-aware tracking controller achieves 100% success across all simulation trajectories and 100% success in real F1-tenth and gate tracking, while a vision-only baseline shows lower success (60% on F1-tenth, 100% on gate) and larger tracking errors during fast out-of-view scenarios. In summary, we introduce FalconTrack , a unified perception and tracking framework for vision-based aerial tracking. Our contributions are threefold: (1) an automated labeling pipeline that generates photorealistic labels for diverse irregular targets without manual annotation; (2) a unified multi-head perception with staged training and reprojection consistency that zero-shot sim-to-real transfers on three geometrically diverse objects and backgrounds; (3) a physics-aware tracking controller with class-conditioned dynamics priors that enables closed-loop visual tracking on a real quadrotor across multiple target types, trajectories, and environments. Fig. 3 : Qualitative Perception in Unseen GSplats and Real World: The first row shows mask and 6-DoF pose estimations on RGB images rendered by FalconGym 2.0 with the target