3D UAV Trajectory Estimation and Classification from Internet Videos via Language Model
Haoxiang Lei, Daotong Wang, Shenghai Yuan, Jianbo Su* · 2026
Read 3D UAV Trajectory Estimation and Classification from Internet Videos via Language Model by Haoxiang Lei, Daotong Wang, Shenghai Yuan, Jianbo Su* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09070.txt.
Introduction
Haoxiang Lei and colleagues map 3D UAV paths and types from web videos without manual labels. Language-driven haul pulls UAV videos while vision-language reasoning picks apt clips, then a training-free cross-modal module infers path guesses and type cues. A physics-informed refinement sets time smoothness and motion consistency, and zero-shot transfer improves more and more with more online video, nearly matching state-of-the-art results.
Catalog record
| CREATOR | Haoxiang Lei, Daotong Wang, Shenghai Yuan, Jianbo Su* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34211 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.09070
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3D UAV Trajectory Estimation and Classification from Internet Videos via Language Model Haoxiang Lei, Daotong Wang, Shenghai Yuan, Jianbo Su* Abstract Reliable 3D trajectory estimation of unmanned aerial vehicles (UAVs) is a fundamental requirement for anti-UAV systems, yet the acquisition of large-scale and accurately annotated trajectory data remains prohibitively expensive. In this work, we present a novel framework that derives UAV 3D trajectories and category information directly from Internet-scale UAV videos, without relying on manual annotations. First, language-driven data acquisition is employed to autonomously discover and collect UAV-related videos, while vision-language reasoning progressively filters task-relevant segments. Second, a training-free cross-modal label generation module is introduced to infer 3D trajectory hypotheses and UAV type cues. Third, a physics-informed refinement process is designed to impose temporal smoothness and kinematic consistency on the estimated trajectories. The resulting video clips and trajectory annotations can be readily utilized for downstream anti-UAV tasks. To assess effectiveness and generalization, we conduct zero-shot transfer experiments on a public, well-annotated 3D UAV benchmark. Results reveal a clear data scaling behavior: as the amount of online video data increases, zero-shot transfer performance on the target dataset improves consistently, without any target-domain training. The proposed method closely approaches the current state-of-the-art, highlighting its robustness and applicability to real-world anti-UAV scenarios. Code and datasets will be released upon acceptance. I Introduction Unmanned aerial vehicles (UAVs) have become increasingly prevalent across civilian and commercial domains, which in turn has intensified concerns regarding airspace safety and counter-UAV defense. A fundamental capability for such systems is the accurate estimation of UAV trajectory in 3D space, as reliable 3D trajectory information is essential for interception planning, and risk-aware decision-making [ 5 , 18 ] . Despite sustained research interest, progress in 3D UAV trajectory estimation remains constrained by the limited availability of large-scale, high-quality annotated 3D UAV data. Most existing approaches for UAV trajectory estimation and classification rely heavily on large amounts of well-annotated data [ 5 , 16 , 18 , 6 ] , whose collection is expensive, time-consuming, and inherently difficult to scale. Although multiple public datasets support 2D UAV trajectory analysis [ 4 , 20 , 14 ] , the absence of depth information significantly constrains their applicability in practical counter-UAV deployments. In contrast, datasets providing full 3D UAV trajectories are exceedingly limited. Among the few available resources, MMAUD [ 19 ] offers high-quality and precise 3D annotations. Nevertheless, its construction depends on costly sensors, such as high-precision LiDAR systems, together with extensive manual annotation efforts, which collectively impose substantial obstacles to the large-scale development of 3D UAV trajectory datasets. Fig. 1 : The comparison between the acquisition of existing Anti-UAV datasets, and our proposed methods for anti-UAV datasets acquisition. The challenge stems from two fundamental factors. First, UAV videos collected from online platforms are highly noisy, frequently including first-person recordings or severe camera motion, which obscures reliable observation of UAV dynamics. Second, acquiring accurate 3D trajectory annotations remains prohibitively expensive. For example, the MMAUD dataset [ 19 ] achieves high-precision labeling through Leica MS60 surveying equipment and multi-sensor systems; however, such configurations require costly hardware, meticulous calibration, and substantial manual effort, rendering them inherently unsuitable for large-scale deployment. Together, these constraints highlight the need for a new approach capable of converting Internet videos into reliable 3D trajectory resources without relying on specialized sensors or human annotation. To address these challenges, a novel framework that converts raw Internet-scale UAV videos into accurate 3D trajectories and UAV type labels, without manual annotations or sensing hardware , is proposed. The framework is guided by three key principles. First, language-driven data acquisition , in which an agentic large language model (LLM) autonomously discovers UAV-related videos and cooperates with vision-language models (VLMs) to progressively remove irrelevant or noisy content. Second, a training-free cross-modal label generation integrates vision–language reasoning with geometric constraints to infer zero-shot 3D trajectory hypotheses and category information. Third, physics-informed refinement is employed to impose temporal consistency and kinematic feasibility through sequential estimation. The resulting video clips and trajectory annotations can be readily utilized for downstream anti-UAV tasks. We validate the framework via zero-shot transfer on the 3D MMAUD benchmark [ 19 ] , which closely approaches the current state-of-the-art. Importantly, the proposed framework demonstrates a clear scaling behavior : as the amount of online video data increases, zero-shot transfer performance on the target dataset improves consistently, without any target-domain training. Our main contributions are summarized as follows: 1. A scalable framework that derives UAV 3D trajectories and category labels directly from Internet-scale videos without manual annotation or costly sensors is proposed, and reveals a clear data scaling behavior. 2. A language-driven data acquisition is designed, in which an agentic LLM works in conjunction with vision–language models to automatically retrieve and refine task-relevant UAV video content. 3. A training-free cross-modal label generation and physics-informed refinement pipeline is introduced that integrates vision–language cues with sequential estimation to ensure temporal coherence and kinematic plausibility. 4. We conduct a comprehensive zero-shot evaluation on the 3D MMAUD benchmark, achieving competitive performance, closely approaching the current state-of-the-art, while establishing a reproducible and scalable paradigm for anti-UAV perception. II Related works II-A Annotated Datasets for Anti-UAV Research Several datasets have been developed for anti-UAV research, including the Anti-UAV benchmarks [ 4 , 21 , 20 ] and DUT Anti-UAV [ 22 ] , which provide large-scale annotated videos for UAV tracking. However, these datasets rely on labor-intensive frame-level labeling and are limited to 2D tracking in RGB or thermal imagery, restricting their applicability to real-world 3D trajectory estimation. To enable 3D annotation, Zheng et al. [ 23 ] proposed a multi-camera UAV dataset using synchronized and calibrated RGB cameras. While this approach supports 3D reconstruction, it incurs high hardware and calibration costs and has not been publicly released, limiting reproducibility. The MMAUD dataset [ 19 ] offers a high-quality 3D benchmark by employing high-precision Leica MS60 ground truth together with LiDAR, stereo cameras, and a microphone array. Nevertheless, its construction depends on expensive multi-sensor systems and meticulous calibration. In summary, existing anti-UAV datasets either focus on 2D tracking with costly manual annotation or rely on complex sensing infrastructures for 3D labeling, posing major challenges for scalable 3D UAV trajectory estimation. Fig. 2 : The overall framework consists of three main parts, the one on the top is the Language-driven Data Acquisition module for high-quality anti-UAV data. The others on the bottom are training-free cross-modal label generation for 3D UAV trajectory and classification, and the physics-informed refinement. The final output is the 3D trajectory of the UAV, and the LiDAR point cloud is used for visualization only, not for processing. II-B Dataset-Driven Trajectory Estimation Approaches Recent studies on anti-UAV trajectory estimation have largely relied on high-quality, well-annotated datasets or sensor-rich acquisition systems. For example, YOLO-RAW [ 8 ] improves UAV detection robustness in adverse weather scenarios, while AV-DTEC [ 16 ] and AV-FDTI [ 18 ] exploit LiDAR-generated pseudo labels to train audio–visual or audio-only models for UAV detection and trajectory inference. Similarly, AAUTE [ 5 ] estimates UAV 3D trajectories from audio signals using LiDAR-based supervision and smoothing techniques, achieving strong performance on the MMAUD dataset. Despite these advances, existing approaches fundamentally depend on curated datasets with precise annotations or sophisticated