M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments

D.A. Yudin · 2025

Read M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments by D.A. Yudin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-17044.txt.

Introduction

Dmitry Yudin surveys methods for building multimodal 3D maps of changing scenes and adds a modular method of his own. The 2025 article serves robotics and self-driving transport, where no shared form for dynamic 3D scenes yet exists. Inputs can span images, point clouds, and text.

A taxonomy sorts current methods by scene type, representation, learning style, and use. That map of the field frames a brief review of recent work. The author's method joins object segmentation and tracking, motion estimation, map building, and data retrieval in linked parts. Uses range from grounding objects in 3D to guiding robot hands. Theory claims back the gains from mixing data modes and foundation models.

Catalog record

CREATORD.A. Yudin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-17044

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments D.A. Yudin Abstract 3D mapping in dynamic environments poses a challenge for modern researchers in robotics and autonomous transportation. There are no universal representations for dynamic 3D scenes that incorporate multimodal data such as images, point clouds, and text. This article takes a step toward solving this problem. It proposes a taxonomy of methods for constructing multimodal 3D maps, classifying contemporary approaches based on scene types and representations, learning methods, and practical applications. Using this taxonomy, a brief structured analysis of recent methods is provided. The article also describes an original modular method called M3DMap, designed for object-aware construction of multimodal 3D maps for both static and dynamic scenes. It consists of several interconnected components: a neural multimodal object segmentation and tracking module; an odometry estimation module, including trainable algorithms; a module for 3D map construction and updating with various implementations depending on the desired scene representation; and a multimodal data retrieval module. The article highlights original implementations of these modules and their advantages in solving various practical tasks, from 3D object grounding to mobile manipulation. Additionally, it presents theoretical propositions demonstrating the positive effect of using multimodal data and modern foundational models in 3D mapping methods. Details of the taxonomy and method implementation are available at https://yuddim.github.io/M3DMap . 1 Introduction In recent years, various foundational neural models for image recognition (e.g., CLIP [ 108 ] , LSeg [ 72 ] , SAM [ 61 ] , DINO [ 99 ] ) and 3D point clouds (Uni3D [ 172 ] , Sonata [ 143 ] , etc.) have rapidly advanced. This has led to the emergence and development of approaches for enriching 3D maps with additional information, such as feature representations of objects aligned with textual modalities, enabling the inclusion of arbitrary objects beyond a fixed set of categories. However, there is no unified approach to representing 3D scenes in such multimodal maps: some works use point clouds/voxels [ 55 , 47 ] , bird’s-eye-view projections [ 167 , 105 ] , while others employ Neural Radiance Fields (NeRF) [ 60 ] , Signed Distance Fields (SDF) [ 149 ] , Gaussian splatting [ 106 ] . A significant body of research is based on representing scenes as various types of graphs [ 42 , 137 , 161 , 140 ] . This diversity is largely due to the wide range of applications, from navigation and control of intelligent robots and autonomous vehicles to 3D/4D scene reconstruction in Virtual/Augmented Reality systems. Moreover, there is a noticeable bias toward methods based on image or video modalities, with significantly less attention paid to point cloud data, despite its growing importance due to the decreasing cost of LiDARs. There are no unified datasets or benchmarks that simultaneously address tasks such as recognition (segmentation), tracking, object description in 3D space, question-answering about dynamic 3D scenes, or using maps for robot planning and control. However, multimodal datasets based on established benchmarks like ScanNet [ 30 ] and 3DScan [ 130 ] are gradually emerging. This work attempts to systematize existing research in multimodal 3D mapping, covering all major scene representations and key applications. Special attention is given to dynamic scenes, where there is a notable lack of research, benchmarks, and technical solutions. Additionally, based on extensive research experience in this field, a methodology is proposed that integrates various computer vision methods into a unified framework for multimodal 3D mapping, adaptable to diverse practical tasks using different data modalities and scene representations. Contributions of the article can be summarized as follows: • A categorization of approaches for constructing multimodal 3D maps, including those for dynamic environments, is proposed. It is detailed in Section 2 . • A modular approach called M3DMap is introduced for building multimodal 3D maps. It uses neural models for object extraction and tracking, trainable odometry estimation, algorithms for constructing accumulated and instantaneous map representations, and methods for retrieving multimodal data (see Section 3 ). Experiments show that the proposed modules significantly improve performance in real-world downstream tasks in robotics and autonomous transportation (see Section 5 ). • Theoretical propositions are presented, demonstrating the improvement in object recognition quality when constructing 3D maps by leveraging multiple modalities or additional encoders for raw sensor data (see Section 4 ). 2 Categorization of Related Approaches This article proposes a taxonomy for systematizing existing research in multimodal 3D mapping, as shown in Figure 1 . Figure 1: Proposed taxonomy of methods for constructing multimodal 3D maps The taxonomy categorizes methods based on four main criteria: • Ability to Take into Account Scene Changes : Two broad categories are considered—static scenes (where all objects are stationary) and dynamic 3D scenes (including three types of dynamics: objects moving within the robot’s field of view, outside it, or interactive objects that can open/close, turn on/off, etc.). • Scene Representation : This criterion extends the classification proposed in [ 156 ] with Neural Implicit Scene Representation, where the map is implicitly formed as an updatable tensor from raw input data. Such representations are typically used for planning and control tasks in robotics. • Usage of Learnable Approaches : This criterion also considers methods that do not use deep learning, often aimed at solving practical robot localization tasks. Distinguishing between modular and end-to-end approaches can be challenging; here, end-to-end methods are defined as single neural models where gradients can flow from output to raw input data during training. • Applications : Seven key practical domains are highlighted. Tasks like 3D Object Grounding include detection and segmentation in 3D scenes, with open-vocabulary approaches dominating as they accommodate natural language queries. Applications related to SLAM, pose estimation, and place recognition are grouped under Vehicle Localization and Mapping. 4D scene reconstruction adds a fourth dimension—time—to spatial coordinates. State Prediction includes approaches that generate maps or estimate the robot’s future state. High-Level Action Planning involves using multimodal 3D map representations with Large Language Models (LLMs) to generate action plans. Low-Level Vehicle Control generates specific robot actions for manipulation or navigation tasks. Question-Answering applications encompass a wide range of subtasks and question types about 3D scenes, including captioning and reasoning. A brief analysis of recent works, particularly those from 2023–2025 with open-source code, is provided below. The primary categorization criterion is the handling of scene dynamics. Dynamic 3D Scenes. Methods in this category are systematized in Tables 1 and 2 . Table 1: Modern Dense Methods for Constructing Multimodal 3D Maps of Dynamic Scenes. The prefix "D:" in scene representation categories stands for Dense Method Year Scene Representation Learn Application Code Map Modalities DynamicGSG [ 39 ] 2025 D: Gaussian Splatting-based Modular Object Grounding, 3D Scene Reconstruction ✓ Gaussians, Image-Text features, Hierarchical 3D Scene Graph 4D-Gaussian-Splatting [ 155 ] 2024 D: Gaussian Splatting-based End-2-End 4D Scene Reconstruction ✓ Color, gaussian features, time 4D-GS [ 139 ] 2024 D: Gaussian Splatting-based End-2-End 4D Scene Reconstruction ✓ Color, gaussian features, time GeFF [ 107 ] 2024 D: NeRF-based End-2-End 3D Scene Reconstruction, Mobile Manipulation ✓ Color, Density, Visual-Language Features Lifelong LERF [ 110 ] 2024 D: NeRF-based End-2-End Object Grounding, 3D Scene Reconstruction CLIP Features, Color Remembr [ 5 ] 2025 D: Neural Implicit Representation End-2-End Object Grounding, High-Level Action

Read the complete text