Real-Time 3D Vision-Language Embedding Mapping

Christian Rauch, Björn Ellensohn, Linus Nwankwo, Vedant Dave, Elmar Rueckert · 2025

Read Real-Time 3D Vision-Language Embedding Mapping by Christian Rauch, Björn Ellensohn, Linus Nwankwo, Vedant Dave, Elmar Rueckert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06291.txt.

Introduction

Christian Rauch and four co-authors present VLEM, a system that lets robots build three-dimensional maps labeled with language. A robot with a depth camera needs maps that are both exact in space and open to plain questions, like pointing out every chair. Recent image-language models can tag pictures with meaning, but folding their output into a live 3D map is hard: it costs computing power and memory, and many systems need known camera poses.

VLEM takes only a raw color-plus-depth stream and merges the image tags into one consistent, exact 3D map in real time. Against three rival systems, it segments scenes better under open-ended labels and keeps its map more compact. The authors also show it working live in robot grasping tasks and in mobile mapping. They present it as a practical step toward robots that understand spaces the way people describe them.

Catalog record

CREATORChristian Rauch, Björn Ellensohn, Linus Nwankwo, Vedant Dave, Elmar Rueckert
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06291

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-Time 3D Vision-Language Embedding Mapping Christian Rauch, Björn Ellensohn, Linus Nwankwo, Vedant Dave, Elmar Rueckert Abstract A metric-accurate semantic 3D representation is essential for many robotic tasks. This work proposes a simple, yet powerful, way to integrate the 2D embeddings of a Vision-Language Model in a metric-accurate 3D representation at real-time. We combine a local embedding masking strategy, for a more distinct embedding distribution, with a confidence-weighted 3D integration for more reliable 3D embeddings. The resulting metric-accurate embedding representation is task-agnostic and can represent semantic concepts on a global multi-room- as well as on a local object-level. This enables a variety of interactive robotic applications that require the localisation of objects-of-interest via natural language. We evaluate our approach on a variety of real-world sequences and demonstrate that these strategies achieve a more accurate object-of-interest localisation while improving the runtime performance in order to meet our real-time constraints. We further demonstrate the versatility of our approach in a variety of interactive handheld, mobile robotics and manipulation tasks, requiring only raw image data. Website: https://rtvlem.github.io . I Introduction Many robotic downstream tasks, such as manipulation and navigation, require a metric-accurate understanding of the 3D environment as well as semantic information about objects and places to interact with. Classically, this problem has been approached by closed-set semantic mapping [ 1 ] , which combines 2D or 3D segmentation with dense 3D reconstruction [ 2 ] , in order to remove dynamic disturbances [ 3 ] , and for robust localisation [ 4 ] . Closed-set segmentation uses a fixed set of classes and labels, limiting the application of these approaches to certain domains and specific query patterns. Some of these limitations can be avoided by implicit rigid motion assumptions [ 5 ] , but are then missing explicit semantics. Recent combinations of Vision and Text Transformers [ 6 ] , trained on large datasets, enable open-set image and text relations within a common embedding space. Since the majority of these models are trained on internet-scale datasets with predominantly 2D images, they are not directly applicable to 3D. For an application relying on a stream of 2D images, this also implies that only the current field-of-view (FoV) can be considered at any point in time, excluding mobile tasks, where objects-of-interest are outside the current FoV. Recent work uses implicit neural representations [ 7 , 8 ] to merge and render embeddings in place of colour channels [ 9 , 10 , 11 ] . Often, these approaches are not real-time capable due to their training regimes requiring multiple dependant stages operating globally on the entire dataset, including ground truth camera poses, instead of a continuous data stream. Furthermore, most implicit representations are only queryable in a synthesised 2D view of rendered embeddings and do not provide a metric-accurate geometric representation. Both these properties are required for an interactive zero-shot localisation of objects-of-interest in most downstream robotic tasks. This work proposes a method for the real-time mapping of image embeddings in 3D for open-set robotic downstream tasks. We only require a stream of colour and depth images to reconstruct an environment with image embeddings ( Figure 1 ). We do not require ground truth camera poses or any pre-training on the target environment. This makes our method environment- and task-agnostic and serves as a building block for other tasks which require a language interface to robotic applications. kitchen coffee machine workshop toolbox table cup Fig. 1 : 3D reconstruction of various environments and their vision-language embeddings in real-time (left), and the response to text queries (right). In summary, this work contributes: • a simple yet versatile method for integrating 2D embeddings in 3D in real-time, requiring only a stream of colour and depth images, • a 3D representation of image embeddings that is queryable via natural language in real-time, • a novel approach for extracting segment-aligned 2D image embeddings and their integration in 3D, • demonstrations of interactive language instructions for specifying objects and areas of interest in handheld, manipulation and mobile robotic settings. II Related Work II-A Vision-Language Models in Robotics In contrast to classic closed-set methods trained on specific labels, novel Vision-Language Models (VLMs) enable the open-set association of images with their text descriptions [ 6 , 12 ] , or other modalities [ 13 ] , via a common embedding space, using individual transformers for image, text, or other modalities. VLMs have been used in robotics for open-set tracking of objects in the current camera FoV [ 14 ] , for interactive pose estimation of relevant parts of tools [ 15 ] , and for navigation via hand-drawn instructions [ 16 ] . By focusing on a single task and the current FoV, these approaches cannot generalise to other tasks or operate on a global level, such as localising tools outside the current FoV. In contrast, we integrate the open-set VLM embeddings in a task-agnostic 3D representation in order to enable a variety of interactive robotic use-cases on the same vision-language representation. II-B Implicit Neural Representations Due to the availability of vast amounts of 2D images and text, Vision Transformers (ViT) are predominantly trained on 2D image data [ 17 ] . In a robotic application, this constrains their use to the current camera FoV. Implicit Neural Representations (INR) or Novel View Synthesis (NVS) methods [ 7 , 8 ] are trained and overfitted on a closed colour image data set with their camera poses, and focus on rendering colour images for novel camera views at a high visual fidelity. Compared to previous reconstruction methods, such as point clouds with oriented splats [ 2 ] , these methods provide a more compact encoding of the training data and more detailed 2D renderings. Since these methods rely only on colour images and do not use depth or true camera poses, they cannot reproduce a scale-accurate metric representation of the environment. Furthermore, their implicit representation has to be converted to a dense Cartesian representation before it can be used in geometric tasks, such as navigation and manipulation. These limitations of implicit representations, together with their training regime, effectively prevents their use in robotics. To avoid these limitations, we propose a metric-accurate direct representation in form of a point cloud, which allows us to directly access the representation for robotic tasks and to store the full dynamic range of the vision-language embeddings. II-C Embedding Integration Implicit Neural Representations are used to render novel views of vision-language embeddings [ 9 , 10 ] or segmentation features [ 18 ] in addition to colour. The same technique has also been applied to traditional point cloud representations [ 11 ] . VLMs that are trained on image and text pairs, such as OpenCLIP [ 19 ] , only generate a single embedding vector per image and text. While this is sufficient for room-level scenes [ 20 ] , it limits the granularity of the representation for object-level tasks. Instead, more discriminative embeddings can be extracted on the individual segments of an image [ 21 ] , enabling more fine-grained responses to queries [ 11 , 10 ] . We show, that a naive implementation of this local embedding strategy can lead to embeddings of neighbouring segments diluting each

Read the complete text