E2E-Fly: An Integrated Training-to-Deployment System for End-to-End Quadrotor Autonomy

Fangyu Sun†, Fanxing Li†, Linzuo Zhang, Yu Hu, Renbiao Jin, Shuyu Wu, Wenxian Yu*, Danping Zou* · 2026

Read E2E-Fly: An Integrated Training-to-Deployment System for End-to-End Quadrotor Autonomy by Fangyu Sun†, Fanxing Li†, Linzuo Zhang, Yu Hu, Renbiao Jin, Shuyu Wu, Wenxian Yu*, Danping Zou* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-12916.txt.

Introduction

Fangyu Sun and seven colleagues unify quadrotor training, validation, and real deployment. Transfer to reality stalls on slow rendering, physics gaps, sensor mismatch, and missing unified platforms. E2E-Fly pairs an agile quadrotor platform with a full workflow. A fast simulator supports differentiable physics and reinforcement learning with structured rewards. Two-stage validation uses simulation transfer plus hardware testing. Policies deploy on two real quadrotors via identification, randomization, latency compensation, and noise modeling. Six control tasks run in the real world.

Catalog record

CREATORFangyu Sun†, Fanxing Li†, Linzuo Zhang, Yu Hu, Renbiao Jin, Shuyu Wu, Wenxian Yu*, Danping Zou*
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81353
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.12916

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

E2E-Fly: An Integrated Training-to-Deployment System for End-to-End Quadrotor Autonomy Fangyu Sun†, Fanxing Li†, Linzuo Zhang, Yu Hu, Renbiao Jin, Shuyu Wu, Wenxian Yu*, Danping Zou* Abstract Training and transferring learning-based policies for quadrotors from simulation to reality remains challenging due to inefficient visual rendering, physical modeling inaccuracies, unmodeled sensor discrepancies, and the absence of a unified platform integrating differentiable physics learning into end-to-end training. While recent work has demonstrated various end-to-end quadrotor control tasks, few systems provide a systematic, zero-shot transfer pipeline, hindering reproducibility and real-world deployment. To bridge this gap, we introduce E2E-Fly, an integrated framework featuring an agile quadrotor platform coupled with a full-stack training, validation, and deployment workflow. The training framework incorporates a high-performance simulator with support for differentiable physics learning and reinforcement learning, alongside structured reward design tailored to common quadrotor tasks. We further introduce a two-stage validation strategy using sim-to-sim transfer and hardware-in-the-loop testing, and deploy policies onto two physical quadrotor platforms via a dedicated low-level control interface and a comprehensive sim-to-real alignment methodology, encompassing system identification, domain randomization, latency compensation, and noise modeling. To the best of our knowledge, this is the first work to systematically unify differentiable physical learning with training, validation, and real-world deployment for quadrotors. Finally, we demonstrate the effectiveness of our framework for training six end-to-end control tasks and deploy them in the real world. I Introduction End-to-end learning has gained increasing attention in quadrotor autonomy by directly mapping raw sensor observations to actuator commands through neural networks [ 10 , 32 , 53 , 17 ] . Unlike conventional modular architectures, which decompose the whole task into sequential modules, such as perception, planning, and motor control, end-to-end policies bypass those intermediate computations, thereby reducing system latency and mitigating error propagation across modules. This data-driven paradigm has demonstrated compelling performance in agile flight [ 53 , 17 ] , obstacle avoidance [ 56 , 39 ] , and high-speed trajectory tracking [ 33 , 9 ] , outperforming traditional model-based controllers in highly dynamic or uncertain environments. Within this paradigm, differentiable physics learning, which leverages analytical gradients of the quadrotor dynamics for efficient policy training, has demonstrated impressive performance in quadrotor tasks such as hovering [ 19 ] and high-speed collision avoidance [ 58 , 20 ] . By providing exact gradients, it substantially improves learning efficiency and final policy quality compared with black-box reinforcement learning (RL) and standard imitation learning approaches. Despite recent successes, the practical application of end-to-end methods to real-world quadrotors is still challenging. Firstly, training an end-to-end control policy involves choosing appropriate simulators and learning algorithms as well as designing effective rewards for the given task. Secondly, quick and safe validation must be particularly required for quadrotor tasks, as crashes can result in severe damage not only for the vehicle but also for the onboard sensors and computers. Finally, accurate sim-to-real alignment is essential for real-world deployment, including system identification, sensory noise modeling, and control latency compensation. All those factors motivate the development of a system that integrates simulation, learning algorithms, and hardware into a cohesive training-to-deployment pipeline. Such a unified system can substantially accelerate the train-to-deployment procedure, improving the efficiency and safety during debugging and evaluating the trained networks. TABLE I: Key characters comparison of state-of-the-art approaches in this area. Type Hardware Render Sensors Algorithm Validation Deployment Number of Tasks Flightmare [ 44 ] simulator \times Unity D,R,S,I RL HIL \times \times RotorS [ 14 ] simulator \times OpenGL D,R,I \times \times \times \times CrazyS [ 43 ] simulator \times OpenGL D,R,I \times \times \times \times CrazySim [ 30 ] simulator \times OpenGL D,R,S,I \times \times \times \times FlightGoggles [ 16 ] simulator \times Unity D,R,S,I \times HIL \times \times FastSim [ 7 ] simulator \times Unity D,R,S,I,L RL HIL \times \times AirSim [ 34 ] simulator \times Unreal D,R,S,I \times HIL \times \times PyBulletDrone [ 37 ] simulator \times OpenGL D,R,S,I RL \times \times \times OmniDrone [ 55 ] simulator \times Isaac Sim D,R,S,I,L RL \times \times \times VisFly [ 29 ] simulator \times Habitat-sim D,R,S,I RL,DS \times \times \times Agilicious [ 11 ] platform \checkmark Unity D,R,S,I \times HIL LC \times AirGym [ 21 ] platform \checkmark Isaac Sim D,R,I RL \times SI,DR 5 E2E-Fly platform \checkmark Habitat-sim D,R,S,I RL,DS HIL,sim-to-sim SI,LC,DR,NM 6 a I, R, D, S, and L refer to IMU, RGB, depth, segmentation, and LiDAR, respectively. HIL refers to hardware-in-the-loop validation. RL and DS represent reinforcement learning and differentiable simulation. SI, LC, DR, and NM represent system identification, latency compensation, domain randomization, and noise modeling, respectively. \checkmark indicates support, \times indicates no support. E2E-Fly is a system-level quadrotor platform that provides the entire technical stack from training to validation and to deployment, including a customization parallel training environment, two hardware platforms for offboard and onboard validation, a fast renderer based on Habitat-sim, four types of sensors, sim-to-sim, and hardware-in-the-loop interfaces, algorithms covering RL and differentiable simulation, and four sim-to-real alignment techniques. Many research platforms for quadrotors have been developed, while only a few of them support training-to-deployment effectively. Agilicious [ 11 ] is a co-designed hardware and software research platform for aggressive autonomous flight, featuring a high thrust-to-weight ratio (5g acceleration) and a powerful onboard GPU-based computer (Nvidia Jetson TX). Its modular software stack, including a rendering-based simulator and hardware-in-the-loop support, provides flexibility for both traditional perception–planning–control pipelines and learning-based methods. However, its primary focus lies in enabling high-performance flight rather than offering a thorough workflow for end-to-end policy training or systematic sim-to-real transfer. AirGym [ 22 ] , in contrast, is specifically designed for RL. It integrates an RL training environment with hardware for validation, employing PX4 for low-level control and an RK3588-based onboard computer. It investigates several practical considerations for successful sim-to-real transfer, such as system identification, action smoothing, and domain randomization. Nevertheless, AirGym does not incorporate differentiable physical learning algorithms and lacks guidance on reward design for common quadrotor tasks and evaluation on different learning algorithms. While these systems are valuable, there remains a need for a unified framework that integrates advanced simulators, state-of-the-art learning algorithms, comprehensive reward design, and training guidelines for common tasks, and detailed protocols for validation and sim-to-real alignment. Such a system would significantly lower the barrier for developing robust end-to-end quadrotor controllers and provide a consistent methodology for both researchers and practitioners. To this end, we present E2E-Fly, a unified training-to-deployment pipeline designed specifically for developing end-to-end control policies for a diverse range of quadrotor tasks. E2E-Fly integrates advanced simulators, state-of-the-art learning algorithms, and comprehensive guidelines for reward design, system identification, and sim-to-real alignment. The platform also provides two reference quadrotor designs for real-world evaluation, together with six representative benchmark tasks that illustrate the full workflow of training, validating, and deploying end-to-end policies using algorithms such as PPO [ 42 ] and Back-Propagation Through Time (BPTT) [ 36 ] . For the training stage, E2E-Fly adopts VisFly [ 29 ] , a high-performance differentiable simulator capable of rendering multi-modal sensory observations at high frame rates while providing end-to-end differentiability through the quadrotor dynamics. This enables the use of highly efficient gradient-based learning techniques such as BPTT [ 36 ] , offering substantially improved sample efficiency compared with black-box RL such as PPO

Read the complete text