Real-Time Object Tracking with On-Device Deep Learning for Adaptive Beamforming in Dynamic Acoustic Environments

Jorge Ortigoso-Narro, Jose A. Belloch, Adrian Amor-Martin, Sandra Roger, Maximo Cobos · 2025

Read Real-Time Object Tracking with On-Device Deep Learning for Adaptive Beamforming in Dynamic Acoustic Environments by Jorge Ortigoso-Narro, Jose A. Belloch, Adrian Amor-Martin, Sandra Roger, Maximo Cobos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-19396.txt.

Introduction

Jorge Ortigoso-Narro and four co-authors build an embedded system that listens where a moving object is. The device joins camera-based tracking with acoustic beamforming, steering a microphone array's focus in real time. Single-camera depth estimates plus stereo vision locate objects in three dimensions.

The array is a flat concentric circle of small MEMS microphones, compact and power-thrifty, steering across both azimuth and elevation. Tracking output updates the focus continuously, so the captured sound follows the target even among competing sources. Tests show clear gains in signal-to-interference ratio, suiting the design for teleconferencing, smart homes, and assistive hearing tools.

Catalog record

CREATORJorge Ortigoso-Narro, Jose A. Belloch, Adrian Amor-Martin, Sandra Roger, Maximo Cobos
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-19396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-Time Object Tracking with On-Device Deep Learning for Adaptive Beamforming in Dynamic Acoustic Environments Jorge Ortigoso-Narro, Jose A. Belloch, Adrian Amor-Martin, Sandra Roger, Maximo Cobos Abstract Advances in object tracking and acoustic beamforming are driving new capabilities in surveillance, human-computer interaction, and robotics. This work presents an embedded system that integrates deep learning–based tracking with beamforming to achieve precise sound source localization and directional audio capture in dynamic environments. The approach combines single-camera depth estimation and stereo vision to enable accurate 3D localization of moving objects. A planar concentric circular microphone array constructed with MEMS microphones provides a compact, energy-efficient platform supporting 2D beam steering across azimuth and elevation. Real-time tracking outputs continuously adapt the array’s focus, synchronizing the acoustic response with the target’s position. By uniting learned spatial awareness with dynamic steering, the system maintains robust performance in the presence of multiple or moving sources. Experimental evaluation demonstrates significant gains in signal-to-interference ratio, making the design well-suited for teleconferencing, smart home devices, and assistive technologies. † † affiliationnotice: Preprint, sumbitted to Journal of Supercomputing. 1 Introduction The increasing demand for intelligent systems that can reliably localize sound sources and capture directional audio has made object tracking and acoustic beamforming essential technologies. Applications range from consumer-grade solutions such as teleconferencing, smart home automation, and assistive devices to mission-critical systems including anti-drone defense, perimeter surveillance, and autonomous security platforms. These systems rely on the integration of spatial awareness and audio enhancement to operate effectively in complex and dynamic environments ( 15 ; 23 ) . Object tracking and computer vision systems based on deep learning have become widely studied and substantially advanced in recent years ( 29 ) . These methods deliver fast and reliable performance, enabling precise identification and spatial localization of objects, along with robust classification capabilities. Compared to audio-only systems, this enables a more intelligent, context-aware approach, enhancing situational awareness and interaction with dynamic environments. However, a key drawback is that the inference part of deep learning models is often computationally demanding, requiring specialized hardware such as Graphics Processing Units (GPUs) or dedicated Artifical Intelligence (AI) accelerators. Achieving real-time performance in such systems necessitates careful optimization and efficient model deployment, particularly in resource-constrained and portable platforms. To enable real-time performance of deep learning–based computer vision models, a range of optimization strategies is employed. Model quantization, for instance, reduces weights and activations from floating-point to lower-precision formats (e.g., INT8), which decreases memory usage and computation time with minimal accuracy loss ( 16 ) . Pruning and architecture simplification further reduce inference latency by eliminating redundant parameters and layers ( 10 ; 14 ) improving the performance (specially in edge AI platforms). In a complementary domain, Micro-Electromechanical Systems (MEMS) microphone arrays provide an efficient foundation for embedded audio systems due to their compact form factor, low power requirements, and ease of integration ( 22 ) . Their small size enables dense array configurations, which in turn support accurate beamforming and spatial localization in constrained environments. These arrays are increasingly adopted in robotics, smart assistants, and surveillance, where precise directional audio capture is required. Combined with modern signal processing hardware, they offer a practical solution for real-time spatial audio processing on embedded platforms. The integration of visual tracking with acoustic beamforming has helped advance sound source localization, allowing for highly precise directional audio capture in both traditional and modern systems. Multimodal devices such as the Microsoft Kinect and Intel RealSense D455 combine RGB cameras, depth sensors, and microphone arrays to spatially focus audio by targeting a speaker’s mouth through 3D depth mapping, effectively reducing background noise. Recent developments, such as the work by Nagasha et al. ( 21 ) , which merges face tracking with beamforming, and mobile robots addressing the cocktail party problem ( 26 ) , leverage deep learning to improve performance in multi-speaker environments. The main contribution of this work is the design and implementation of a compact, energy-efficient embedded system that integrates deep learning–based 3D object localization with dynamically steerable MEMS microphone beamforming, enabling real-time directional audio capture in dynamic acoustic environments. Thus, we integrate hardware and software changes to improve the overall performance. 2 Proposed system The proposed system consists of three primary components: (1) a visual perception module with depth estimation capabilities, utilizing either a stereo or monocular camera, (2) a planar concentric circular MEMS microphone array for audio acquisition, and (3) an NVIDIA Jetson Orin Nano (8GB version) embedded processor that handles tracking, classification, and beam steering. These components are integrated through a low-latency pipeline that facilitates closed-loop interaction between the visual and acoustic subsystems. Figure 1 illustrates the high-level block diagram of the proposed system architecture. Figure 1: High-level block diagram design. 2.1 Vision Module To provide visual information to the system, a low-cost USB stereo camera (GXIVISION-LSM22100) was selected as the sensing hardware. The vision module therefore consists of two synchronized image sensors integrated in a single stereo unit. The joint resolution of the stereo pair is 2560\times 720 pixels, i.e., each camera provides frames at a resolution of 1280\times 720 pixels. The camera streams MJPEG-compressed video at 30 FPS over USB, which offers a cost-effective solution while still providing sufficient spatial resolution and frame rate for real-time object tracking and depth estimation. 2.1.1 Object classification Object classification has been a key focus in computer vision research. Early approaches leaned on carefully designed features, like Scale Invariant Feature Transform (SIFT) and Histograms of Oriented Gradients (HOG), paired with simple classifiers such as Support Vector Machines ( 20 ; 4 ; 3 ) . These methods worked well in specific, controlled environments but often faltered when faced with challenges like changing light conditions, partial object obstruction, or busy backgrounds. With the rise of deep learning, Convolutional Neural Networks (CNNs) have taken center stage, delivering impressive gains in both accuracy and reliability across a wide range of image datasets. Models like AlexNet, VGG, and ResNet stand out for their ability to automatically learn layered patterns from raw images, eliminating the need for handpicked features. These architectures paved the way for more advanced systems, and architectures adapting to diverse real-world scenarios with greater ease ( 12 ; 28 ) . In recent advancements, the YOLO (You Only Look Once) series has gained traction for its efficiency in object classification and detection. Renowned for its rapid processing and dependable performance, YOLO supports real-time applications while maintaining strong accuracy. For our research, we selected YOLOv11, which refines earlier versions with sharper architectural enhancements, such as the C3k2 block and C2PSA attention mechanism, alongside optimized training techniques. These improvements enable YOLOv11 to reduce the number of parameters and maintain fast inference times. This model delivers a well-balanced solution, ensuring swift computation and consistent outcomes for our classification tasks ( 17 ) . 2.1.2 Depth Estimation To correctly estimate the direction of arrival of each source from the visual data, its 3D position in space is required. Depth estimation plays a central role in this process by providing the missing third dimension (labeled in this work as z-axis) to the 2D image-plane coordinates extracted from object detection. One common approach to estimating depth is through stereopsis, which leverages a calibrated stereo camera setup with a known baseline B and

Read the complete text