Audio-3DVG: Unified Audio - Point Cloud Fusion for 3D Visual Grounding
Duc Cao-Dinh Khai Le-Duc, Anh Dao Bach Phan Tat Chris Ngo, Duy M. H. Nguyen Nguyen X. Khanh Thanh Nguyen-Tang · 2025
Read Audio-3DVG: Unified Audio - Point Cloud Fusion for 3D Visual Grounding by Duc Cao-Dinh Khai Le-Duc, Anh Dao Bach Phan Tat Chris Ngo, Duy M. H. Nguyen Nguyen X. Khanh Thanh Nguyen-Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00669.txt.
Introduction
Duc Cao-Dinh and seven co-authors teach machines to find an object in a three dimensional scene from spoken words, in a paper from 2025. Past work used typed text, while speech based search stayed little explored and hard.
Their frame splits the job in two. First it names which objects the speech mentions at all. Then a guided focus step weighs target candidates against those named objects to cut clutter errors. The team also made spoken forms of three standard test sets. Their method set a new best mark for speech based search and rivaled text based tools.
Catalog record
| CREATOR | Duc Cao-Dinh Khai Le-Duc, Anh Dao Bach Phan Tat Chris Ngo, Duy M. H. Nguyen Nguyen X. Khanh Thanh Nguyen-Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 88 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00669
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Audio-3DVG: Unified Audio - Point Cloud Fusion for 3D Visual Grounding Duc Cao-Dinh Khai Le-Duc, Anh Dao Bach Phan Tat Chris Ngo, Duy M. H. Nguyen Nguyen X. Khanh Thanh Nguyen-Tang Abstract 3D Visual Grounding (3DVG) involves localizing target objects in 3D point clouds based on natural language. While prior work has made strides using textual descriptions, leveraging spoken language—known as Audio-based 3D Visual Grounding—remains underexplored and challenging. Motivated by advances in automatic speech recognition (ASR) and speech representation learning, we propose Audio-3DVG, a simple yet effective framework that integrates audio and spatial information for enhanced grounding. Rather than treating speech as a monolithic input, we decompose the task into two complementary components. First, we introduce Object Mention Detection, a multi-label classification task that explicitly identifies which objects are referred to in the audio, enabling more structured audio-scene reasoning. Second, we propose an Audio-Guided Attention module that captures interactions between candidate objects and relational speech cues, improving target discrimination in cluttered scenes. To support benchmarking, we synthesize audio descriptions for standard 3DVG datasets, including ScanRefer, Sr3D, and Nr3D. Experimental results demonstrate that Audio-3DVG not only achieves new state-of-the-art performance in audio-based grounding, but also competes with text-based methods—highlighting the promise of integrating spoken language into 3D vision tasks. (*) (*) footnotetext: Equal contribution 1 Introduction Visual grounding (VG) of referring expressions—the task of identifying visual entities described in natural language—has made significant progress in the 2D computer vision domain [ 54 , 59 , 46 , 84 ] . With the rapid advancement of 3D sensing technologies and spatial data representations, this task has naturally extended into the 3D domain, where spatial reasoning becomes increasingly crucial. Unlike 2D images composed of grid-aligned pixels, 3D data—typically represented as point clouds—encodes richer geometric and spatial structures. This shift introduces both novel opportunities and unique challenges for accurately grounding language in three-dimensional space. In line with this evolution, recent studies have transitioned from grounding objects in 2D images [ 53 , 11 , 15 , 70 ] to grounding in 3D scenes [ 10 , 1 ] , where the goal is to localize objects referenced by natural language within a point cloud. While these advancements have yielded strong results, most approaches remain reliant on textual input. This dependence poses a barrier to practical deployment, as it requires users to manually input referring expressions using keyboards or touchscreens—a process that is inefficient in hands-busy or eyes-busy situations and inaccessible for users with motor impairments. Figure 1 : a) Application of Audio based 3DVG to robot navigation task. b) Example sentences referring to objects in 3D scenes. Green boxes indicate objects of the same category as the target, while blue boxes highlight relational objects mentioned in the speech. (a) Audio-based 3D Visual Grounding enables object localization to support robot navigation. (b) The chair has its back to the windows . the chair is next to the desk in the corner of the room facing the brown armchair . To overcome the limitations of text-only inputs, recent research has begun exploring audio-based 3D visual grounding. A pioneering example is AP-Refer [ 96 ] , a multimodal framework that replaces text with spoken language as the input modality. This framework aligns raw point clouds with corresponding audio signals to localize objects mentioned in natural speech, enabling audio-driven robot navigation, as illustrated in Figure 1 a. Despite its innovation, AP-Refer exhibits two critical limitations. First, it lacks an effective attention mechanism for fusing audio and spatial features, resulting in limited cross-modal understanding. Second, it ignores relational objects mentioned in speech, relying solely on audio and individual object features for grounding. This approach is especially inadequate in densely populated scenes where spatial context is vital. In this paper, we present a novel framework that addresses both issues and significantly narrows the performance gap between audio-based and text-based methods. Our contributions are threefold: To address the first limitation, we build on the observation that target objects are often spatially related to other instances of the same category and to additional objects explicitly mentioned in the spoken input (as shown in Figure 1 b). We introduce an auxiliary task called Object Mention Detection , which aims to identify the presence of relational objects referenced in the utterance. These relational objects serve as spatial anchors that guide the model in identifying the correct target among candidates. To resolve the second limitation, we propose an Audio-Guided Attention Module , which learns spatial and semantic relationships between candidate objects and relational entities, all conditioned on the audio signal. This attention mechanism improves the model’s ability to focus on relevant spatial dependencies, enhancing localization performance. In addition, we contribute new benchmark datasets for the 3DVG-Audio task. These include high-quality synthetic speech datasets based on existing 3DVG benchmarks, as well as a real-world spoken dataset to evaluate generalization. As shown in our experiments, our model achieves substantial improvements over previous audio-based methods, reaching 17.03% and 17.15% accuracy on Acc@25 and Acc@50, respectively, while maintaining competitive performance with text-based systems. In summary, the key contributions of this paper are as follows: • We introduce a new framework for Audio 3D Visual Grounding that incorporates both target proposals and relational objects, effectively reducing noise in cluttered point clouds. • We design a novel Audio-Guided Attention module that captures semantic and spatial relationships conditioned on spoken input. • We create standardized benchmark datasets for the 3DVG-Audio task, including both synthetic and real-world audio, to facilitate robust evaluation and comparison. • Our model establishes new state-of-the-art results among audio-based methods and achieves performance comparable to leading text-based approaches, highlighting its strong generalization ability and computational efficiency. 2 Related Work 2.1 Multi-modal research based on audio Audio, a ubiquitous and easily accessible modality, has been extensively studied since the early days of Artificial Intelligence [ 78 ] , particularly in tasks such as audio classification [ 23 , 24 , 51 ] . With the rapid advancement of deep learning in recent years, there has been growing interest in integrating audio with other modalities to address real-world challenges. Notable areas of multi-modal research involving audio include audio-text fusion [ 55 , 89 , 87 ] . These audio-based multi-modal approaches commonly rely on pre-trained audio feature extraction modules to effectively capture meaningful audio representations. 2.2 3D Visual Grounding 3D grounding aims to identify the object in a 3D scene that is referred to by a natural language expression. Datasets [ 1 , 10 ] and methods [ 10 , 12 , 101 , 86 , 99 ] have been proposed to address this challenging task. Existing approaches can generally be categorized into two groups: one-stage and two-stage frameworks. One-stage methods directly fuse text features with visual representations at the patch or point level to regress the target bounding box [ 56 , 44 ] , enabling flexible detection of various objects from the input sentence. In contrast, two-stage methods follow a detect-then-match paradigm [ 92 , 12 , 99 , 86 ] , where the first stage generates object proposals and the second stage selects the best match based on the input description. This decoupling of object perception and cross-modal matching makes two-stage methods more interpretable and easier to analyze.