Audio Array-Based 3D UAV Trajectory Estimation with LiDAR Pseudo-Labeling
Allen H.-X. Lei, Tianchen Deng, Han Wang, Jianfei Yang, Shenghai Yuan* · 2024
Read Audio Array-Based 3D UAV Trajectory Estimation with LiDAR Pseudo-Labeling by Allen H.-X. Lei, Tianchen Deng, Han Wang, Jianfei Yang, Shenghai Yuan* (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12698.txt.
Introduction
Allen Lei and four co-authors teach a sound-based system to track drones in three dimensions. Their 2024 paper answers a safety need as small drones crowd the skies. Microphone arrays hear a drone but cannot alone place it well. The team uses LiDAR as a temporary teacher, then drops it.
LiDAR scans estimate each flight path without human labels. Those paths become stand-in labels for training the audio network. Sound is first turned into mel-spectrograms, a picture of pitch over time. After training, the system predicts paths from sound alone. A smoothing process sharpens the tracks, and the method topped a standard drone dataset.
Catalog record
| CREATOR | Allen H.-X. Lei, Tianchen Deng, Han Wang, Jianfei Yang, Shenghai Yuan* |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12698
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio Array-Based 3D UAV Trajectory Estimation with LiDAR Pseudo-Labeling Thanks: This work is supported by National Research Foundation, Singapore, under its Medium-Sized Center for Advanced Robotics Technology Innovation. Allen H.-X. Lei, Tianchen Deng, Han Wang, Jianfei Yang, Shenghai Yuan* Abstract As small unmanned aerial vehicles (UAVs) become increasingly prevalent, there is growing concern regarding their impact on public safety and privacy, highlighting the need for advanced tracking and trajectory estimation solutions. In response, this paper introduces a novel framework that utilizes audio array for 3D UAV trajectory estimation. Our approach incorporates a self-supervised learning model, starting with the conversion of audio data into mel-spectrograms, which are analyzed through an encoder to extract crucial temporal and spectral information. Simultaneously, UAV trajectories are estimated using LiDAR point clouds via unsupervised methods. These LiDAR-based estimations act as pseudo labels, enabling the training of an Audio Perception Network without requiring labeled data. In this architecture, the LiDAR-based system operates as the Teacher Network, guiding the Audio Perception Network, which serves as the Student Network. Once trained, the model can independently predict 3D trajectories using only audio signals, with no need for LiDAR data or external ground truth during deployment. To further enhance precision, we apply Gaussian Process modeling for improved spatiotemporal tracking. Our method delivers top-tier performance on the MMAUD dataset, establishing a new benchmark in trajectory estimation using self-supervised learning techniques without reliance on ground truth annotations. I Introduction In recent years, Unmanned Aerial Vehicles (UAV) [ 1 , 2 , 3 ] have become widely accessible [ 4 , 5 ] , raising significant security concerns such as privacy violations [ 6 , 7 ] , unauthorized surveillance [ 8 , 9 , 10 , 11 ] , and risks in restricted airspace [ 12 , 13 , 14 ] , including threats to civilian aviation. Their use in cross-border drug smuggling [ 15 ] and military operations [ 16 , 17 ] underscores the urgent need for effective UAV tracking technologies [ 18 ] . Despite the pressing need for UAV tracking systems, existing studies predominantly rely on visual information [ 19 , 20 , 21 , 22 , 23 , 24 , 25 ] for 2D tracking or 3D detection from RADAR/LiDAR [ 26 , 27 ] [ 28 , 29 , 7 , 30 , 31 , 32 , 33 , 34 , 35 ] . However, these approaches are limited by poor visibility, restricted field of view, and the high cost of RADAR/LiDAR for large-area surveillance and low-altitude smuggling UAV. The utilization of audio data, despite its accessibility and cost-effectiveness, remains largely unexplored [ 36 , 37 , 38 ] in UAV trajectory estimation [ 39 , 40 ] . Given the affordability and ubiquity of microphones compared to LiDAR and cameras [ 41 , 42 , 43 , 44 ] , there is a significant opportunity to develop innovative 3D tracking solutions [ 45 , 46 , 47 , 48 , 49 ] that leverage audio information. The main challenges in audio-based UAV 3D trajectory estimation are data annotation and filtering out ambient noise, which is particularly difficult in real-world scenarios where annotated data is not readily available. To address this, we propose an Audio Array-Based 3D UAV Trajectory Estimation with LiDAR to generate unsupervised pseudo-ground truth labels for self-supervised learning. During deployment, only audio is used, with Gaussian Process smoothing applied to ensure continuous trajectory estimation. Our contributions are summarized as follows: Fig. 1: System Motivation and Overview. Fig. 2: Proposed Self-Supervised Framework. 1. We present a novel self-supervised Teacher-Student Network for audio vectoring-based 3D trajectory estimation, enabling robust UAV tracking across diverse lighting conditions without manual labels. 2. We propose a cost-effective unsupervised method for 3D drone trajectory estimation using LiDAR scans as pseudo-ground truth for the Audio Perception Network. 3. We benchmarked our method against various SOTA approaches, achieving the highest accuracy in inferring drone 3D trajectories. 4. We are open-sourcing our work to benefit the community. https://github.com/AllenLei666/AAUTE . II Proposed Method We introduce a novel framework for Unmanned Aerial Vehicle (UAV) trajectory estimation, which is combined with a Teacher Network and a Student Network, as illustrated in Fig. 2 . For the Student Network, the input consists of audio waves collected by the microphone vector. The microphone vector captures audio information pertaining to the UAV. The process begins with converting the audio waves into audio spectrograms to represent the temporal and spectral characteristics of the sound. Next, an encoder is utilized to extract features from the audio spectrograms, capturing both temporal and frequency-domain information. Then, the Gaussian Process Smoothing is implemented to smooth the estimated trajectory. For the Teacher Network, two types of LiDAR data are processed and fused using an unsupervised learning method to estimate the 3D trajectory of UAVs. This high-precision trajectory estimate serves as a reference for the subsequent self-supervised learning phase. The LiDAR-estimated trajectory serves as pseudo labels to train the framework, enabling accurate UAV trajectory prediction from audio alone, with results cross-checked against ground truth data. II-A Unsupervised Teacher Network The Livox Mid360 provides a 360° horizontal field of view and a 59° vertical field of view above the ground, capturing both background and UAV point clouds up to 40 m. Since the LiDAR data contains significant noise, our main goal is to remove the noise and extract the UAV’s LiDAR data. Initially, we need to isolate the UAV point clouds from the background. Based on recent approaches [ 28 ] , we use 20 consecutive LiDAR scans as a chunk and apply the DBSCAN clustering algorithm with eps = 1.0 and min-samples = 10 to group the points into clusters for best performance. The connected component analysis is then implemented to merge adjacent clusters with distance-threshold = 2.5 and min-component-size = 20 for best-extracting performance to form larger coherent regions. The largest and smallest clusters are removed as they are typically considered noise and partial background, respectively. To further refine the UAV point cloud, we analyze the density of each cluster and select the cluster corresponding to the UAV’s point cloud. The DJI Livox Avia LiDAR, with its 70-degree conical upward-facing field of view, tracks UAV movement up to 350m. To process the Avia data, we use the farthest point sampling technique to reduce the number of points while preserving the spatial distribution of the UAV point cloud. The dynamic part of the Livox Mid360 point cloud is then extracted and fused with the downsampled Avia data to form the complete UAV point cloud. DBSCAN is applied again to remove any remaining solar noise. Finally, with the complete UAV point cloud, B-Spline fitting is utilized to interpolate a smooth curve through the 3D UAV point cloud, generating pseudo labels that represent the UAV’s trajectory with the mean error of 0.5 m compared to the real UAV’s trajectory. II-B Self-Supervised Audio Feature Learning To process the audio information, we begin by collecting audio waves from a vector of microphone. The audio signals are then segmented into 2-second clips and sampled at a rate of 48,000 samples per second. Subsequently, the audio segments are converted into mel-spectrograms using a sweep window size of 2048 and a hop length of