CAVER : Curious Audiovisual Exploring Robot
Luca Macesanu, Boueny Folefack, Samik Singh, Ruchira Ray, Ben Abbatematteo, Roberto Martín-Martín · 2025
Read CAVER : Curious Audiovisual Exploring Robot by Luca Macesanu, Boueny Folefack, Samik Singh, Ruchira Ray, Ben Abbatematteo, Roberto Martín-Martín (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07619.txt.
Introduction
Luca Macesanu, Boueny Folefack, and Samik Singh built CAVER, a robot that learns objects by sight and sound together. A shaken bottle looks and sounds different from a tapped one, and those linked cues reveal material and contents. Most robots never gather such paired experience on their own.
CAVER explores objects actively, building rich audiovisual representations as it pokes and listens. Curiosity guides which interactions to try next, so knowledge grows efficiently. Skills like sorting by material or copying a tune played by ear become plausible. The robot shows how self-driven play can teach cross-modal facts that passive datasets miss.
Catalog record
| CREATOR | Luca Macesanu, Boueny Folefack, Samik Singh, Ruchira Ray, Ben Abbatematteo, Roberto Martín-Martín |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CAVER : Curious Audiovisual Exploring Robot Luca Macesanu, Boueny Folefack, Samik Singh, Ruchira Ray, Ben Abbatematteo, Roberto Martín-Martín Abstract Multimodal audiovisual perception can enable new avenues for robotic manipulation, from better material classification to the imitation of demonstrations for which only audio signals are available (e.g., playing a tune by ear). However, to unlock such multimodal potential, robots need to learn the correlations between an object’s visual appearance and the sound it generates when they interact with it. Such an active sensorimotor experience requires new interaction capabilities, representations, and exploration methods to guide the robot in efficiently building increasingly rich audiovisual knowledge. In this work, we present CAVER , a novel robot that builds and utilizes rich audiovisual representations of objects. CAVER includes three novel contributions: 1) a novel 3D printed end-effector, attachable to parallel grippers, that excites objects’ audio responses, 2) an audiovisual representation that combines local and global appearance information with sound features, and 3) an exploration algorithm that uses and builds the audiovisual representation in a curiosity-driven manner that prioritizes interacting with high uncertainty objects to obtain good coverage of surprising audio with fewer interactions. We demonstrate that CAVER builds rich representations in different scenarios more efficiently than several exploration baselines, and that the learned audiovisual representation leads to significant improvements in material classification and the imitation of audio-only human demonstrations. https://caver-bot.github.io/ I Introduction Humans learn and exploit multimodal audiovisual cues in everyday life to obtain a more complete understanding of their environment and broader manipulation capabilities. We routinely fuse audio and vision to understand materials and reproduce behaviors: tapping a mug reveals glass vs. ceramic, and hearing a melody lets a musician find the right key. Building similar capabilities in robots would increase their robustness and autonomy, but requires a representation that couples how things look with how they sound when interacted with , and a way to acquire that representation efficiently through interaction. Fig. 1 : Curiously Building and Exploiting an Audiovisual Representation with CAVER . CAVER incrementally builds a KNN-based audiovisual representation capable of audio-to-visual and visual-to-audio prediction. To explore an environment, CAVER considers many candidate interaction points (dots), and ranks their uncertainty by comparing their visual features to those of prior samples. CAVER then collects an audio sample of the most uncertain point (red) using our novel impact tool. The audio and visual features are added as a pair to the audiovisual representation in an intrinsically motivated process that results in efficient interactive exploration. Underlying the multimodal audiovisual abilities of humans is a rich developmental process in which children interact with and explore the physical world around them [ 49 ] . When children play with objects, they generate multimodal stimuli from which they learn audiovisual correlations in a self-supervised manner [ 53 ] . Their exploration is incremental and, critically, curious : it is driven to prioritize unfamiliar experiences so as to quickly and efficiently learn the audiovisual properties of the world [ 35 , 20 ] . While prior work in machine learning and robotics has explored how to build similar audiovisual representations [ 46 , 5 , 7 , 11 ] , they rely on previously acquired large datasets of visual and audio pairs collected manually, largely failing to account for how a robotic agent might explore an environment autonomously and efficiently. In this work, we introduce CAVER , C urious A udio v isual E xploring R obot, a robot that autonomously builds and exploits an audiovisual representation of objects (Fig. 1 ). CAVER integrates a novel, custom-built impact tool that enables a robot to “tap” on object surfaces to elicit an audio response. CAVER draws inspiration from human curiosity to guide the robot through interaction with uncertain objects, first actively exploring visually unfamiliar objects to learn correlations between their appearance and impact acoustics, then using those correlations for downstream tasks. Concretely, a strategy of uncertainty-guided sampling chooses where to strike next to generate a visual [ 51 , 14 ] and auditory [ 26 ] feature representation of the interaction point, while a simple but effective k-nearest-neighbor (KNN) methodology on the audiovisual features enables immediate retrieval in either direction (vision \rightarrow audio and audio \rightarrow vision) to exploit the information for manipulation (see Sec. IV-D ). We test CAVER by exploring various household scenes and incrementally building its audiovisual representation, and evaluate it on downstream tasks including audio prediction, material classification, and audio-based imitation. Across several household environments, CAVER enables rapid learning of object visual and acoustic properties in an autonomous and self-supervised manner. Through our experiments, we demonstrate that CAVER ’s curious, uncertainty-driven exploration strategy yields a marked improvement in audio prediction quality over several sampling and exploration alternatives, and its audiovisual representation achieves 87% accuracy in material classification, 66% accuracy in audio-based imitation of melodies, and 42% accuracy in action recognition (compared to a 27% accurate human baseline). Fig. 2 : Overview of CAVER ’s audiovisual representation and curious exploration. CAVER curiously and efficiently learns correlations between object visual appearance and acoustic properties. Given candidate hitting points, CAVER uses a KNN model with fine tuned features from foundation vision models to predict corresponding impact sounds. To select informative interaction points, CAVER selects the most uncertain candidate hitting location using distance in visual feature space between a candidate (red) and all prior samples (green) as a proxy for uncertainty. After sampling the most uncertain candidate, the corresponding visual and audio embeddings are paired as two sides of the Audio-Visual representation. This can best be thought of as a bi-directional mapping where audio features can be used to predict visual features, visual features can be used to predict audio features, and concatenating the embeddings gives an informative multimodal representation of that sample point. In summary, in CAVER our contributions include: 1) a reproducible, 3D-printed impact end-effector ; 2) a multi-scale audiovisual representation combining visual and audio features in a KNN model, and learned feature weights aligning visual to audio distance; 3) an uncertainty-guided exploration policy that prioritizes visually distinct object parts using farthest-first selection; and 4) a demonstration that this representation is immediately useful across tasks—audio prediction, material classification, musical imitation, and audio-only manipulation inference—without large-scale pretraining on paired data. II Related Work CAVER is a robot that can perceive, curiously explore, and quickly learn an audiovisual representation that can predict sound based on images and be applied to solve audio-related downstream tasks. In this section, we contrast it to representative prior work that explored multimodal robot learning, interactive perception, and action sound generation. Multimodal and Audiovisual Robot Learning. Leveraging multi-modal data in robotic tasks is less explored than the conventional vision-based learning setting [ 30 , 25 , 57 , 47 , 22 ] . While several methods have combined vision with tactile information [ 4 , 16 , 29 ] or force feedback [ 40 , 28 , 32 ] for increased performance, leveraging audiovisual signals remains relatively underexplored. Only a few methods have explored leveraging audio in imitation learning [ 34 , 10 , 34 ] , to handle occlusions [ 8 ] , or as supervision for skill learning [ 58 ] . This primarily stems from a lack of large annotated audio datasets,