A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception

Weitung Chen, Phil Tinn, Per Gunnar Auran, Martin Ludvigsen, Peter Halland Haro · 2026

Read A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception by Weitung Chen, Phil Tinn, Per Gunnar Auran, Martin Ludvigsen, Peter Halland Haro (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01398.txt.

Introduction

Chen and co-authors release a sonar-visual dataset from a Norwegian fjord. Underwater robots pair cameras with sonar for perception. Vision gives rich semantic detail about a scene. Sonar gives robust range data where cameras struggle. Cross-modal prediction between them lacks paired training data. SOVIS holds over 76,000 paired frames for that gap. Seventeen dives at six Trondheimfjord sites produced the frames.

A pipeline cleans and synchronizes the sensor streams. An interactive tool speeds labeling of paired data. A proof-of-concept task tests cross-modal fish detection. It uses just a small labeled subset of the data. Detection beats a monocular camera baseline by seven times. The dataset opens paths like dense sonar prediction from images.

Catalog record

CREATORWeitung Chen, Phil Tinn, Per Gunnar Auran, Martin Ludvigsen, Peter Halland Haro
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01398

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Sonar-Visual Dataset for Cross-Modal Underwater Robot Perception Thanks: This work was supported by the Research Council of Norway under Project No. 328193. Thanks: 1 Weitung Chen is with the Massachusetts Institute of Technology (MIT), Cambridge, MA, USA (email: [email protected] ). Thanks: 2 Phil Tinn, Per Gunnar Auran, and Peter Halland Haro are with SINTEF, Trondheim, Norway (email: [email protected] , [email protected] , and [email protected] ). Thanks: 3 Martin Ludvigsen is with the Department of Marine Technology, Norwegian University of Science and Technology (NTNU), Trondheim, Norway (email: [email protected] ). Weitung Chen, Phil Tinn, Per Gunnar Auran, Martin Ludvigsen, Peter Halland Haro Abstract Underwater robots typically use both cameras and sonar for perception to leverage the rich semantic details of vision and the robust range measurements of acoustics. However, learning to map between these modalities via cross-modal prediction remains underexplored due to limited sonar-visual paired datasets. We present SOVIS, a sonar-visual dataset for cross-modal underwater perception. SOVIS comprises over 76,000 paired frames collected across 17 dives at six sites in the Trondheimfjord, supported by an end-to-end pipeline that cleans and synchronizes the cross-modal sensor data. We also introduce an interactive annotation tool designed to accelerate the labeling process for this paired data. Finally, we demonstrate a proof-of-concept cross-modal fish detection task using a small subset of labeled data, achieving a 7 × 7\times improvement in [email protected] over a monocular camera baseline. SOVIS serves as the first step toward advancing cross-modal underwater perception research, enabling research directions such as dense sonar prediction from monocular images. AUV autonomous underwater vehicle ROV remotely operated vehicle FLS forward-looking sonar SLAM simultaneous localization and mapping FoV field of view CTD conductivity-temperature-depth IMU inertial measurement unit SD standard deviation MSE mean squared error RMSE root mean squared error MAE mean absolute error NN neural network CNN convolutional neural network SSIM structural similarity index GPS global positioning system SOT single object tracking TABLE I: Comparison of Underwater Perception Datasets. Dataset Sensing Modalities Data Collection Platform Environment Primary Focus SUIM [ 2 ] Mono Cam AUV/Diver Ocean Semantic Segmentation NKSID [ 3 ] Oculus FLS ROV Ocean Object Classification DRACo-SLAM [ 8 ] Oculus FLS + Cam BlueROV2 Mixed (Tank, River) Multi-Robot SLAM RGBS50 [ 5 ] Oculus FLS + Mono Cam Fixed Platform Pool Single Object Tracking (SOT) SOLAQUA [ 9 ] Multibeam + Mono/Stereo Cam BlueROV2 Ocean Aquaculture Inspection SOVIS (Ours) Multibeam Sonar + Mono Cam Blueye X3 Ocean Cross-Modal Detection I Introduction Underwater robots perceive the environment through two different sensing modalities: optical cameras and acoustic sonars. Cameras capture rich semantic information (color, texture, shape) that enables fine-grained recognition at close range, while multibeam sonars provide range measurements through acoustic reflections that remain robust to turbidity and low light. Neither modality alone provides a complete picture of the underwater scene. Cameras lack depth information and degrade rapidly in murky water, whereas sonars deliver sparse echo intensities without the semantic content needed for object-level understanding. If a robot could learn the mapping between these two representations, it would gain capabilities that neither sensor could provide alone. For example, the robot could perform depth estimation from a monocular image, or recognize specific objects just from acoustic sonar returns. Such cross-modal prediction has driven major advances in terrestrial robotics, where learning to translate between LiDAR and camera views [ 7 , 6 ] has become foundational for autonomous driving. However, in the underwater world, this direction remains largely unexplored. The primary bottleneck is the lack of paired multi-modal data in underwater robotics. Unlike driving datasets that can be collected at scale on public roads, underwater data collection requires specialized robotic platforms, dive operations, and manual synchronization of sensors with fundamentally different coordinate systems. Furthermore, simulated data cannot easily close this gap: the complex physics of underwater environments, including acoustic multipath, visual scattering, and depth-dependent sound speed variations, are difficult to model accurately. Fig. 1: SOVIS overview. A Blueye X3 ROV equipped with a monocular camera (green FoV) and an Oculus M750d multibeam sonar (blue FoV) collects synchronized underwater data. (a) Camera frame showing fish right above the seafloor. (b) Corresponding sonar return of the same scene. To address this challenge, we introduce SOVIS (SOnar-VISual), a real-world sonar-visual dataset designed for cross-modal underwater perception (Fig. 1 ). We deploy a Blueye X3 ROV equipped with a monocular camera and a multibeam sonar in the Trondheimfjord, collecting over 76,000 synchronized sonar-visual pairs alongside with temperature and pressure readings. Beside the dataset, we also present a processing pipeline, an interactive cross-modal annotation tool that leverages camera-based detections to accelerate sonar labeling, and a proof-of-concept cross-modal fish detection task using a model trained on a small annotated subset. This paper presents the following contributions: • Sonar-Visual Dataset: We present SOVIS, a real-world sonar-visual dataset collected in the ocean, with an end-to-end processing pipeline that cleans and synchronizes camera and sonar data. Fig. 1 shows how we collected SOVIS and an example visual-sonar pair data. • Cross-Modal Annotation Tool: We develop an interactive labeling interface that renders synchronized camera and sonar views side-by-side, automatically projects annotations between visual image frames and sonar frames. • Cross-Modal Fish Detection: Using a small annotated subset produced with our tool, we demonstrate a proof-of-concept cross-modal fish detection model that achieves 46.7% [email protected], a 7\times improvement over a monocular camera baseline, demonstrating one cross-modal task enabled by the dataset. II Related Work Existing underwater robot perception datasets can be separated into three categories: camera-only, acoustic-only, and multi-modal. Camera-only datasets such as SUIM [ 2 ] provide pixel-level semantic segmentation labels for visually-guided underwater robots, while acoustic-only collections like NKSID [ 3 ] focus on forward-looking sonar data from ROV deployments for object classification. Multi-modal datasets pair these sensing modalities for specific downstream tasks. For example, DRACo-SLAM [ 8 ] pairs Oculus imaging sonars with cameras on BlueROV2 platforms for multi-robot underwater SLAM, RGBS50 [ 5 ] offers time-aligned sonar-camera sequences for single object tracking but was collected entirely in a controlled pool with artificial targets rather than open-water environments, and SOLAQUA [ 9 ] pairs multibeam sonar with mono and stereo cameras on a BlueROV2 for aquaculture inspection. We compare these datasets with SOVIS in Table I . However, in all three categories, a critical gap persists: none of these datasets are designed for cross-modal prediction , where models learn to map between visual and acoustic representations to allow one modality to infer the other. Existing multi-modal datasets generally treat vision and acoustics as parallel inputs for tasks like SLAM, tracking, or reconstruction, leaving cross-modal prediction largely unexplored. In terrestrial robotics and autonomous driving, cross-modal prediction has already driven major advances: methods such as Lift-Splat-Shoot [ 10 ] and BEVFormer [ 6 ] produce bird’s-eye-view representations from camera images alone for 3D detection and planning, while BEVFusion [ 7 ] fuses LiDAR and camera features into a shared bird’s-eye-view representation that enables each modality to inform the other. Crucially, these advances were enabled by large-scale paired datasets such as nuScenes [ 1 ] that provide dense cross-modal correspondences for training. In the underwater domain, no analogous paired dataset with cross-modal tooling exists, leaving this research direction unexplored. SOVIS addresses this gap directly. Beyond providing synchronized sonar-visual data, SOVIS contributes a

Read the complete text