DiffusionCinema: Text-to-Aerial Cinematography Conference: ; ; CCS: Human-centered computing Natural language interfaces CCS: Computer systems organization Robotic autonomy CCS: Computing methodologies Motion path planning CCS: Computing methodologies Neural networks

Valerii Serpiva, Artem Lykov, Jeffrin Sam, Aleksey Fedoseev, Dzmitry Tsetserukou · 2026

Read DiffusionCinema: Text-to-Aerial Cinematography Conference: ; ; CCS: Human-centered computing Natural language interfaces CCS: Computer systems organization Robotic autonomy CCS: Computing methodologies Motion path planning CCS: Computing methodologies Neural networks by Valerii Serpiva, Artem Lykov, Jeffrin Sam, Aleksey Fedoseev, Dzmitry Tsetserukou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17412.txt.

Introduction

Valerii Serpiva and four co-authors present DiffusionCinema, a drone filming system driven by words. A user states the shot in plain language. The system plans the flight path to match. No hand piloting is needed. A sample request asks the drone to orbit a person and reveal a scene. Diffusion models read such prompts and shape the path.

Good aerial shots take skill that most users lack. A word-driven system puts that skill in reach. It could serve creators, news teams, and event staff. The work joins language models with flight control in one loop. It treats camera motion as part of the prompt, not an afterthought.

Catalog record

CREATORValerii Serpiva, Artem Lykov, Jeffrin Sam, Aleksey Fedoseev, Dzmitry Tsetserukou
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17412

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DiffusionCinema: Text-to-Aerial Cinematography Conference: ; ; CCS: Human-centered computing Natural language interfaces CCS: Computer systems organization Robotic autonomy CCS: Computing methodologies Motion path planning CCS: Computing methodologies Neural networks Valerii Serpiva, Artem Lykov, Jeffrin Sam, Aleksey Fedoseev, Dzmitry Tsetserukou Abstract . We propose a novel Unmanned Aerial Vehicles (UAV) assisted creative capture system that leverages diffusion models to interpret high-level natural language prompts and automatically generate optimal flight trajectories for cinematic video recording. Instead of manually piloting the drone, the user simply describes the desired shot (e.g., “orbit around me slowly from the right and reveal the background waterfall”). Our system encodes the prompt along with an initial visual snapshot from the onboard camera, and a diffusion model samples plausible spatio-temporal motion plans that satisfy both the scene geometry and shot semantics. The generated flight trajectory is then executed autonomously by the UAV to record smooth, repeatable video clips that match the prompt. User evaluation using NASA-TLX showed a significantly lower overall workload with our interface ( M = 21.6 M=21.6 ) compared to a traditional remote controller ( M = 58.1 M=58.1 ), demonstrating a substantial reduction in perceived effort. Mental demand ( M = 11.5 M=11.5 vs. 60.5 60.5 ) and frustration ( M = 14.0 M=14.0 vs. 54.5 54.5 ) were also markedly lower for our system, confirming clear usability advantages in autonomous text-driven flight control. This project demonstrates a new interaction paradigm: text-to-cinema flight, where diffusion models act as the “creative operator” converting story intentions directly into aerial motion. Figure 1. DiffusionCinema converts a user’s text prompt and an initial image into a UAV trajectory for autonomously recording a cinematic video. The VLN model interprets the prompt into flight actions, the diffusion model generates the video sequence, visual odometry extracts the synthetic camera movement, and the UAV follows the planned path. 1. Introduction Autonomous camera drones capable of following high-level verbal instructions and capturing cinematic aerial footage have long been envisioned in science fiction. Today, UAVs are widely used in film and media production ( 16 ) , enabling dynamic viewpoints difficult to achieve from the ground. However, producing high-quality aerial footage remains challenging due to limited time and personnel, while requiring efficient trajectories and consistent, meaningful visual framing. Currently, this burden falls on skilled First-Person View (FPV) pilots, who navigate drones while framing shots. Mastery of FPV control ( 18 ) requires extensive training to achieve stable high-speed flight, obstacle avoidance, and accurate 3D positioning, while cinematic storytelling demands expertise in composition, motion, timing, and transitions. Recent advances in AI-driven drone control ( 7 ; 20 ; 9 ) can automate high-speed maneuvers and complex navigation with minimal pilot input. The combination of these skill sets is rare but essential for professional UAV cinematography ( 15 ) , making high-end aerial filming largely inaccessible to non-experts despite affordable hardware. Recent research in language-conditioned UAV control has demonstrated promising mechanisms for abstracting low-level piloting. Prior work includes language-driven planning and control systems ( 26 ; 4 ; 25 ; 21 ) , Vision Language Action (VLA) models that map visual observations and text directly to control commands ( 23 ; 14 ; 2 ; 1 ; 10 ) , and aerial Vision Language Navigation (VLN) frameworks that ground instructions in 3D environments ( 13 ; 24 ; 12 ; 27 ; 5 ) . The diffusion model jointly predicts visual traversability and feasible trajectories from a single RGB image, enabling prompt-free, embodiment-agnostic navigation ( 28 ) . These approaches focus on navigation accuracy, efficiency, or task completion, but not on the stylistic and temporal qualities of cinematic footage. Video diffusion models, trained on large-scale human-shot videos, provide strong generative priors for realistic motion, transitions, and cinematic style. Recent work demonstrates their utility for UAV and robotics control by generating trajectories aligned with realistic drone motion ( 23 ; 14 ) . This makes diffusion models appealing for aerial cinematography, as they encode human-like camera behavior without requiring explicit supervision on filming rules. By treating the diffusion model as a cinematographic prior within the control loop, DiffusionCinema enables text-driven aerial filming without manual piloting. In contrast to prior language-driven UAV systems focused on navigation or efficiency, our approach explicitly targets cinematic quality by inheriting stylistic camera motion from large-scale human-shot video data. This paradigm substantially lowers the barrier for aerial content creation while providing professionals with a tool for rapid prototyping of complex shots. 2. Related Work 2.1. Language-Driven Aerial Autonomy Natural language has emerged as an effective high-level interface for UAV control, abstracting low-level piloting into semantic commands. Early work integrated Large Language Models (LLM) into task planning pipelines that translate verbal goals into executable robotic actions ( 26 ) . This direction evolved into real-time multilingual voice control systems ( 4 ) and LLM-based reasoning frameworks for adaptive UAV and swarm trajectory planning in dynamic environments ( 25 ) . Transformer-based vision-language systems further advanced this paradigm. UAV-VLA ( 21 ) , for example, generates large-scale aerial mission plans directly from satellite imagery and textual descriptions, significantly accelerating mission design. While these approaches demonstrate the feasibility and accessibility of language-conditioned UAV control, they primarily produce waypoint-based plans and do not model continuous camera motion or visual style. 2.2. Vision-Language-Action Models VLA models unify perception, language understanding, and control in end-to-end architectures ( 2 ; 1 ; 10 ) . In aerial robotics, RaceVLA ( 23 ) maps FPV observations and language instructions directly to continuous control commands, while CognitiveDrone ( 14 ) augments this paradigm with reasoning modules for cognitively complex tasks. Despite their effectiveness for navigation and task execution, existing VLA models are trained on utilitarian objectives and do not encode cinematographic principles. Consequently, they lack the implicit priors required for expressive camera motion and visual storytelling. 2.3. Aerial Vision-Language Navigation VLN addresses the grounding of natural language in embodied visual environments. AerialVLN ( 13 ) introduced a benchmark for instruction-following UAVs in large-scale 3D spaces. Subsequent work expanded this setting to city-scale environments ( 12 ; 27 ) and realistic UAV dynamics with multimodal LLM guidance ( 24 ) . Semantic Topo-Metric Representations further enhance spatial reasoning for aerial navigation ( 5 ) . These VLN frameworks emphasize the goal completion and path efficiency, yet they do not model cinematic objectives such as shot continuity, framing, or aesthetic camera motion. 2.4. Video Diffusion Models as Motion Priors Video diffusion models now dominate high-fidelity video generation ( 17 ) , producing long, coherent sequences with realistic camera motion. Trained on large corpora of human-recorded footage, they implicitly encode cinematographic patterns such as tracking shots, orbits, and reveals. FlightDiffusion ( 22 ) demonstrated that diffusion models can generate FPV video trajectories and associated UAV states as motion priors for control policy learning. DiffusionCinema applies diffusion-based video synthesis as a cinematographic prior for real-time UAV trajectory generation, enabling language-driven aerial cinematography grounded in human-shot videos. Figure 2. Generated video sequence for the task “Make a cinematic shot for my video vlog with me as the host.” The top row of images shows frames generated by the DiffusionCinema model, while the bottom row shows the corresponding real footage captured by the UAV following the host. Generated video

Read the complete text