Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls
Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang, Mubarak Shah, Yu Tian · 2026
Read Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls by Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang, Mubarak Shah, Yu Tian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19728.txt.
Introduction
Al Radi and team present Aero-World, turning a pretrained image-to-video model into a controllable aerial generator. Foundation video models impress visually yet train mainly on language rather than control signals. That gap bites in aerial flight, where motion spans free 6-DoF space and small errors cause drift. Aero-World injects acceleration and angular velocity through an action-token stream, supervised by a Physics Probe trained on video-IMU pairs. AeroBench measures action alignment and physical consistency, and Aero-World improves mean alignment from 57 point 7 to 63 point 6.
Catalog record
| CREATOR | Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang, Mubarak Shah, Yu Tian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26869 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19728
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Aero-World: Action-Conditioned Aerial Video Generation from Inertial Controls Abdul Mohaimen Al Radi, Kunyang Li, Yuzhang Shang, Mubarak Shah, Yu Tian Abstract Foundation video models produce visually impressive results, but their use in embodied AI remains limited because they are primarily trained on natural language rather than low-level control signals. This limitation is especially pronounced for aerial flight, where motion occurs in unconstrained 6-DoF space and small errors in ego-motion can produce large trajectory drift. Generating aerial videos that follow fine-grained inertial actions can support scalable training and evaluation of aerial agents by providing a controllable proxy for real-world or expensive simulation data. To address this problem, we propose Aero-World , a method for converting a pretrained image-to-video diffusion model into a controllable aerial video generator. Aero-World injects sequences of translational acceleration and angular velocity into a pretrained latent diffusion transformer through an action-token stream. A frozen latent-space Physics Probe, trained independently on real video–IMU pairs, provides differentiable inertial-consistency supervision during LoRA finetuning while avoiding computationally expensive video decoding. We further propose AeroBench , a benchmark for evaluating whether generated drone videos adhere to low-level action signals. AeroBench uses Action Alignment Score (AAS) to measure agreement with commanded inertial actions and Physical Consistency Rate (PCR) to measure temporal motion stability. On AeroBench, Aero-World improves mean AAS from 57.7 to 63.6 over action-only finetuning and gives a stronger quality-control trade-off than AirScape, with lower FVD (596.5 vs. 1058.6), higher SSIM (0.595 vs. 0.505), and higher Flow-IMU correlation (0.44 vs. 0.20). These results suggest that frozen Physics Probe supervision is a practical mechanism for adapting pretrained video generators toward more action-aligned aerial motion. 1 Introduction Recent large-scale video diffusion models exhibit impressive perceptual realism 33 ; 9 ; 19 ; 38 ; 29 . These models hold the potential to become simulators for embodied agents, which can be utilized for learning policies. However, in order to be a reliable proxy to real-world data or expensive simulations, these models need to possess action-faithfulness and physical consistency, where they are currently lacking 39 ; 37 . This gap is more acute in aerial videos because an aerial object like drone operates in unconstrained 6 degree-of-freedom (6-DoF) space, and the translational and rotational dynamics of drones interact non-linearly 20 . Small errors in roll or pitch induce large spatial drift, and aggressive maneuvers amplify temporal inconsistency. A video model which can produce action-faithful aerial videos thus provides a critical benefit: the ability to safely and scalably train robust control policies entirely in software, reducing cost of operating in extreme environments. Figure 1 : (a) Standard Approach: Language-conditioned video models generate plausible motion without explicit action grounding. (b) Aero-World: injects latent-space physics supervision and 6-DoF IMU conditioning into a pretrained diffusion backbone. (c) The result is stable, action-faithful aerial motion, even in out-of-distribution scenes. In order to solve this, existing literature has introduced simple actions like forward, backwards, left, and right 5 ; 13 ; 1 ; 10 . But this framing has one issue, where there is no way to increase the actuation forces, meaning there is no way to rapidly increase or decrease speed in the ego-motion videos, which is prevalent in most drone flights. Another direction is to add camera parameters, optical flow, pose/depth guidance 15 ; 35 ; 32 to steer the video generation. However, these controls often (i) rely on intermediate annotations or sensors not available to an embodied agent during training 11 , (ii) do not match the agent’s native control interface 31 , and (iii) typically lack a training-time objective that explicitly enforces action-motion consistency of ego-motion 12 . Recent literature has also confirmed that scaling up data and the model does not correlate well with action adherence 16 ; 24 . Obtaining annotations for aerial videos is also very difficult and costly. This is why most pretraining corpora consists primarily of captioned videos rather than paired control signals such as linear acceleration and angular velocity. The low-level 6-DoF inertial signals required for control, namely linear acceleration and angular velocity, are rarely observed as supervision in natural video corpora 23 ; 2 ; 30 . Thus, scaling primarily improves visual realism, not low-level controllability, motivating explicit action-consistency supervision during adaptation. To summarize, we have identified generating aerial videos adhering to control signals is an important challenge for embodied AI. We aim to solve this problem leveraging the existing visual knowledge of video generation models. Rather than retraining a video generation model from scratch, we propose a physics probe-regularized conversion mechanism that reshapes the latent dynamics of a pretrained video diffusion model through inertial-consistency supervision. Because inertial trajectories describe platform-level motion rather than drone-specific motor commands, they provide a control interface that is less tied to a particular actuator model. This also addresses the lack of fine-grained control like rapidly increasing or reducing speed. Working in latent space also means there is no need for heavy decoding during finetuning. Our key idea is to introduce a frozen physics probe that estimates 6-DoF inertial signals (linear acceleration, and angular velocity) from generated video latents, providing a data-driven surrogate for enforcing motion consistency during finetuning. During finetuning, the generator is encouraged to produce motion that matches the commanded inertial sequence by penalizing discrepancies between probe-inferred motion and the input actions. Importantly, the probe is trained independently on real video–IMU pairs and remains frozen during generator adaptation, providing a stable supervision signal without modifying the pretrained DiT backbone. This mechanism can be attached to any existing diffusion video models with minimal architectural changes, enabling them to generate IMU-conditioned aerial videos while preserving the visual priors learned from large-scale video pretraining. Our approach, illustrated in Fig. 2 , instantiates this conversion on a pretrained diffusion video backbone. We train on synchronized frames with IMU sequences from drone flight datasets 4 ; 7 , where high angular velocities and rapid vertical transitions stress temporal coherence. By injecting action embeddings and applying latent-space inertial-consistency regularization, the model learns to respect 6-DoF motion constraints while preserving perceptual realism inherited from large-scale pretraining. We evaluate this framework on our proposed AeroBench , a new benchmark designed for action-conditioned aerial generation. To quantify controllability, we introduce the Action Alignment Score (AAS) , which measures bin-level agreement between commanded IMU actions and probe-inferred motion, and the Physical Consistency Rate (PCR) , which captures frame-to-frame oscillatory instability. Our contributions are: • Inertia-Guided Adaptation: A method for adapting a pretrained diffusion video backbone for IMU-conditioned aerial video generation using latent-space inertial supervision. • Physics Probe Supervision: A frozen differentiable probe that encourages action–motion consistency during lightweight LoRA finetuning. • AeroBench and Metrics: A benchmark with AAS and PCR for evaluating 6-DoF action alignment and temporal stability in action-conditioned aerial video generation. 2 Related Work Foundation Video Models. Large-scale video generation has rapidly advanced through diffusion and transformer-based architectures, producing long, high-fidelity samples from text and image prompts 33 ; 9 ; 19 ; 29 . Recent systems emphasize improved temporal consistency, stronger video autoencoders, and scalable denoising transformers, enabling high-quality image-to-video generation at large model sizes. However, these models are primarily designed for visual realism and prompt adherence ; they typically provide only coarse control signals like text, style 33 , or sparse