Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals

Xiaoxuan Liang, Hong Zhou, Zhaolong Wei, Yansong Li, Shujian Yu, Jeremy Gummeson · 2024

Read Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals by Xiaoxuan Liang, Hong Zhou, Zhaolong Wei, Yansong Li, Shujian Yu, Jeremy Gummeson (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11325.txt.

Introduction

Rebuilding a person's 3D shape from a photo works well in bright rooms and fails in the dark or behind obstacles. Xiaoxuan Liang and co-authors add ears to the eyes: their system, SonicMesh, also listens with ultrasound. Echoes are turned into rough range-and-direction pictures by a radar-style imaging method borrowed from aircraft.

Sound pictures are blurry, so the system first maps body joints from both senses into one shared 3D frame. A two-stage network then fuses the sound and sight features into a full body mesh. In tests covering normal light, dim light, blocked views, and even no line of sight, SonicMesh beat camera-only and radio-based rivals. The authors present it as the first framework to marry sound and vision for this task.

Catalog record

CREATORXiaoxuan Liang, Hong Zhou, Zhaolong Wei, Yansong Li, Shujian Yu, Jeremy Gummeson
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11325

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sonicmesh: Enhancing 3D Human Mesh Reconstruction in Vision-Impaired Environments With Acoustic Signals Xiaoxuan Liang, Hong Zhou, Zhaolong Wei, Yansong Li, Shujian Yu, Jeremy Gummeson Abstract 3D human mesh reconstruction (HMR) from RGB images often degrades under poor illumination, occlusion, and non-line-of-sight conditions. Acoustic sensing provides complementary spatial cues but suffers from low spatial resolution. We propose SonicMesh , which, to the best of our knowledge, is the first acoustic–visual framework for robust 3D human mesh reconstruction. SonicMesh first converts ultrasonic echoes into range–azimuth acoustic images through an Inverse Synthetic Aperture Radar (ISAR)-based imaging process. It then introduces a cross-dimensional anatomical registration module that maps modality-specific 2D joint features into a common canonical 3D human space. The registered anatomical representations are further integrated with acoustic and visual features through a two-stage fusion network for final mesh reconstruction. Experiments demonstrate that SonicMesh achieves accurate and robust 3D human reconstruction across normal, poor-light, occluded, and non-line-of-sight environments, consistently outperforming existing RGB-, radio-frequency (RF)-, and mmWave-based approaches under challenging sensing conditions. † † address: 1 UMass, Amherst 2 Devz AI 3 Eastern Institute of Technology, Ningbo 4 UIC 5 VU Amsterdam 1 Introduction Human mesh reconstruction (HMR) from RGB images remains challenging under adverse visual conditions such as poor illumination, occlusion, and non-line-of-sight (NLOS) settings, where body observations can become incomplete or unreliable. In addition, monocular RGB images inherently suffer from depth ambiguity, further complicating accurate 3D body reconstruction. Reliable HMR is important for applications such as AR/VR [ 17 ] , human pose estimation [ 24 ] , and human–computer interaction [ 15 ] . To improve robustness beyond purely visual sensing, recent studies have explored alternative modalities, including mmWave [ 21 , 4 ] , radio-frequency (RF) sensing [ 5 , 23 ] , and LiDAR [ 8 , 1 ] . Although these sensing technologies provide complementary geometric or depth information, they often rely on specialized hardware and can incur additional deployment cost or power consumption. In contrast, acoustic sensing offers an attractive alternative due to its low cost, ease of deployment, privacy preservation, and robustness to adverse visual conditions [ 9 , 20 ] . Owing to the relatively slow propagation speed of sound, acoustic signals can also provide fine-grained ranging information, with millimeter-level positioning accuracy [ 7 ] . Existing studies have demonstrated the potential of acoustic sensing for spatial perception, including acoustic imaging [ 12 ] and echo-based tracking or gesture recognition [ 7 ] . These properties make acoustic sensing a promising complementary modality for robust HMR when RGB observations are incomplete or unreliable. To the best of our knowledge, existing acoustic sensing methods have not yet demonstrated full 3D human mesh reconstruction, primarily due to environmental noise, signal attenuation, weak reflections, and the limited spatial resolution of acoustic images [ 9 ] . Motivated by these limitations, we propose SonicMesh , which, to the best of our knowledge, is the first end-to-end acoustic–visual framework for full 3D human mesh reconstruction. SonicMesh converts ultrasonic echoes into range–azimuth acoustic images, aligns acoustic and visual joint features in a common 3D anatomical space, and then performs global–local multimodal fusion for mesh reconstruction. In summary, our contributions include: • We propose SonicMesh , the first acoustic–visual framework for full-body 3D human mesh reconstruction using commodity acoustic sensing. • We transform ultrasonic echoes into range–azimuth acoustic images and design a cross-dimensional anatomical registration mechanism that aligns acoustic and visual joint features in a shared common 3D space. • We develop a two-stage acoustic–visual fusion framework that integrates global body structure and local anatomy-aware features, enabling robust HMR under poor illumination, occlusion, and NLOS conditions. 2 Proposed Method 2.1 Acoustic Human-Body Imaging Acoustic sensing characterizes the body through sound reflections from different anatomical regions. We therefore first transform the received acoustic echoes into a two-dimensional spatial representation, referred to as an acoustic image . Specifically, we adopt an Inverse Synthetic Aperture Radar (ISAR)-based acoustic imaging scheme [ 9 ] , where the human subject moves relative to a stationary speaker–microphone pair, forming a synthetic aperture across consecutive chirps, while the body is modeled as a collection of acoustic scatterers. We transmit a sequence of linear frequency-modulated (LFM) chirps. For a chirp duration T_{c} , carrier frequency f_{c} , and bandwidth B , the transmitted signal is given by: s(t)=\operatorname{rect}\!\left(\frac{t}{T_{c}}\right)\cos\!\left[2\pi\left(f_{c}t+\frac{k}{2}t^{2}\right)\right],\hskip 20.00003ptk=\frac{B}{T_{c}}, (1) where k denotes the chirp rate and \operatorname{rect}(\cdot) is the rectangular window function. In our implementation, the transmitted frequency sweeps from 18 kHz to 22 kHz. The transmitted chirp is reflected by the human body and captured by a microphone. For an echo with propagation delay \Delta\tau , its corresponding propagation distance is D=\Delta\tau\times v_{s}/2 , where v_{s} is the sound speed in air. Different body parts therefore produce reflections with different delays and amplitudes, providing spatial cues for human-body imaging. The received echoes are organized into an M\times N profile matrix, where each of the N consecutive profiles contains M samples along the range dimension. We then apply the Hilbert transform and dechirp each profile to obtain the intermediate-frequency (IF) matrix, followed by a 2D FFT along the range and azimuth dimensions, mapping each physical scatterer to a corresponding response in the acoustic image I^{a} . For a scatterer located at (p_{r},p_{a}) , its response in the range–azimuth acoustic image can be expressed as: I^{a}(i_{a},i_{r})=A\,\operatorname{sinc}\left[\frac{2kT}{v_{s}}(i_{r}-p_{r})\right]\operatorname{sinc}\left[\frac{2\theta}{\lambda}(i_{a}-p_{a})\right], (2) where A is a constant determined by the signal amplitude and chirp duration, v_{s} denotes the speed of sound, \lambda is the acoustic wavelength, \theta denotes the effective target rotation angle during imaging, and i_{r} and i_{a} are the indices along the range and azimuth dimensions, respectively. Additional details are provided in the Appendix A . Figure 1 : Overview of the proposed acoustic–visual framework for 3D human mesh reconstruction, including acoustic imaging, cross-dimensional anatomical registration, and multimodal fusion. 2.2 Cross-Dimensional Anatomical Registration Before cross-dimensional registration, we first localize the 2D anatomical keypoints in both modalities. For the RGB image, a standard 2D keypoint detector is used to obtain the J=16 anatomical joints. Due to the lower resolution and more diffuse boundaries of acoustic images, we adopt a two-stage localization strategy for the acoustic modality. Specifically, Fast R-CNN [ 6 ] is first used to detect the human region, followed by a Gaussian mixture model (GMM) to segment the acoustic responses into body-related regions. The joint locations are then estimated from the geometric centers and high-density regions of the corresponding segments (see Appendix B for details), yielding \mathcal{K}^{m}=\{k_{j}^{m}\}_{j=1}^{J},\hskip 20.00003ptk_{j}^{m}\in\mathbb{R}^{2},\hskip 10.00002ptm\in\{a,v\}. (3) For each modality m\in\{a,v\} , we employ a modality-specific HRNet [ 16 ] to extract a dense feature map: F^{m}=\operatorname{HRNet}_{m}(I^{m}). (4) For the j -th detected anatomical keypoint k_{j}^{m}\in\mathcal{K}^{m} , the corresponding local feature is sampled from F^{m} as: z_{j}^{m}=\operatorname{Sample}(F^{m},k_{j}^{m})\in\mathbb{R}^{d}. (5) We define a canonical 3D anatomical space based on the SMPL-X body template [ 14 ] and uniformly sample a 20\times 20\times 20 grid within its bounding volume, yielding N=8000 candidate locations \mathcal{C}=\{c_{i}\}_{i=1}^{N} , where c_{i}\in\mathbb{R}^{3} . Here, c_{i} denotes a canonical candidate location, while K_{j}^{*} denotes the ground-truth 3D position of the j -th joint in the same canonical coordinate system. Each candidate location is embedded into the same feature space through an MLP, h_{i}=g(c_{i})\in\mathbb{R}^{d}. (6) We then measure the cosine

Read the complete text