Building temporally coherent 3D maps with VGGT for memory-efficient Semantic SLAM
Gergely Dinya, Péter Halász, András Lőrincz, Kristóf Karacs, Anna Gelencsér-Horváth · 2025
Read Building temporally coherent 3D maps with VGGT for memory-efficient Semantic SLAM by Gergely Dinya, Péter Halász, András Lőrincz, Kristóf Karacs, Anna Gelencsér-Horváth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-16282.txt.
Introduction
Gergely Dinya and four co-authors present a system that builds three-dimensional maps with object meaning attached, fast enough for near real-time use such as helping a person navigate. Their pipeline rests on VGGT, a transformer that infers scene geometry from images, but VGGT alone uses too much memory for a continuous video stream.
The system processes images in a sliding window and aligns the local submaps into one growing map. It reuses the VGGT tracking head to lift two-dimensional instance masks into three-dimensional objects. It stores timestamps and object identities so changes in a room can be spotted later. Tests on standard benchmarks and on custom assistive navigation data show the approach holds up in real-world scenes while staying memory efficient.
Catalog record
| CREATOR | Gergely Dinya, Péter Halász, András Lőrincz, Kristóf Karacs, Anna Gelencsér-Horváth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-16282
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Building temporally coherent 3D maps with VGGT for memory-efficient Semantic SLAM Gergely Dinya, Péter Halász, András Lőrincz, Kristóf Karacs, Anna Gelencsér-Horváth Abstract We present a fast, spatio-temporal scene understanding framework based on Visual Geometry Grounded Transformer (VGGT). The proposed pipeline is designed to enable efficient, close to real-time performance, supporting applications including assistive navigation. To achieve continuous updates of the 3D scene representation, we process the image flow with a sliding window, aligning submaps, thereby overcoming VGGT’s high memory demands. We exploit the VGGT tracking head to aggregate 2D semantic instance masks into 3D objects. To allow for temporal consistency and richer contextual reasoning the system stores timestamps and instance-level identities, thereby enabling the detection of changes in the environment. We evaluate the approach on well-known benchmarks and custom datasets specifically designed for assistive navigation scenarios. The results demonstrate the applicability of the framework to real-world scenarios. Figure 1 : Overview of the proposed pipeline: memory-efficient, temporally coherent semantic SLAM with VGGT. 1 Introduction 3D scene understanding underpins assistive navigation: users must operate in cluttered, unfamiliar, and evolving indoor environments. A practical system therefore needs a temporally coherent 3D map that stays stable across viewpoint change, partial occlusions, and genuine scene state changes. Moreover, approaches should be designed for a low memory footprint and high speed. Feed-forward multi-view models such as VGGT [ 1 ] provide fast depth and pose estimation. However, applying this paradigm to long sequences is challenging for several reasons: (i) memory requirements grow rapidly with sequence length; (ii) temporal dependencies are not modeled explicitly; and (iii) the target application demands sustained spatio-temporal consistency across hundreds of frames while also incorporating semantics — i.e., handling open-world objects — and adapting to rearrangements and genuine scene changes. In realistic deployments, input arrives as a continuous video stream and the map must be updated online. Methods that rely on global, post-hoc optimization over full sequences presuppose offline access to all data and are ill-suited for real-time guidance. Accordingly, we target an online, streaming-friendly formulation that incrementally maintains geometry and semantics as frames arrive. 1 1 1 We will publish the code in a public repository. We focus on indoor environments, a primary deployment setting for assistive navigation, where mapping is challenged by reflective and transparent surfaces, repeated structures, motion blur from handheld sensor devices, and frequent occlusions. We primarily target a monocular RGB setup; when auxiliary depth from commodity mobile sensors (e.g., LiDAR) is available, we use it only to stabilize metric scale, and we report results for both configurations. Beyond integrating semantics into a geometrically accurate 3D map suitable for SLAM, the semantic layer should provide a firm basis to perform tasks with real user impact; accordingly, we consider the broadly relevant problem of finding an empty seat in indoor environments as a representative challenge. This can be challenging in unfamiliar spaces and because seat state can change even when the surrounding scene appears static. Our contributions can be summarized as follows: • We introduce a memory-efficient, block-wise VGGT pipeline that enables long-sequence 3D reconstruction in a streaming setting, overcoming the prohibitive memory growth of standard feed-forward multi-view models while preserving geometric consistency. • We develop a temporally coherent semantic mapping strategy that fuses 2D instance segmentation with the VGGT tracking head, producing stable, persistent 3D object identities across viewpoint changes, occlusions, and re-entries into the field of view. • We propose a lightweight change-detection mechanism based on timestamped object trajectories and depth-based visibility reasoning, allowing the system to detect object presence, disappearance, and state changes in realistic indoor environments. • We integrate these components into an online SLAM framework that maintains both geometry and semantics incrementally, supports real-time operation on commodity hardware, and is well suited for assistive navigation tasks such as identifying free or occupied seats. • We provide quantitative evaluations on standard datasets and qualitative results on custom assistive-navigation scenarios, demonstrating that the method achieves competitive accuracy while remaining computationally efficient during inference. 2 Related work Recent years have seen rapid progress in close to real time 3D scene representations from motion. We build on VGGT [ 1 ] , a pretrained, feed-forward vision transformer that, from RGB images alone, predicts depth, a point cloud, and camera intrinsics and poses (extrinsics). The authors benchmark on an NVIDIA H100 GPU and report 8.75s for a 200-frame sequence. The memory footprint exceeds 40GB at 200 frames and increases approximately linearly with sequence length on this setup. VGGT Long [ 2 ] extends VGGT’s memory-limited, locally consistent point-map generation to kilometer-scale scenes. It partitions the sequence into overlapping chunks ( \sim 60-75 frames), aligns each chunk to the previous via Iteratively Reweighted Least Squares (IRLS) optimization, and mitigates drift through loop detection and Sim(3) alignment of overlapping pointclouds, followed by a global optimization over the chunk transforms. Because it requires multiple passes and access to the full sequence—including a final global adjustment over all frames—the method is not suited to streaming input; moreover, dynamic objects are typically down-weighted and effectively treated as outliers due to low confidence. FastVGGT [ 3 ] accelerates VGGT via training-free token merging in the Global Attention module, achieving a 4 \times speed-up on an NVIDIA H100 GPU. Whereas VGGT, provided 80GB VRAM, runs out of memory at around 300 frames, FastVGGT increases the feasible sequence length to over 1\,000 frames ( \simeq 3.5\times ). To address true streaming/online use, StreamVGGT [ 4 ] is designed to process incoming frames incrementally. However, because its spatio-temporal tokens scale with sequence length, both memory usage and latency grow with the number of frames. It replaces global self-attention with temporal causal attention and caches historical keys/values as an implicit memory. The cached-token design implies that memory footprint grows with sequence length, which the authors explicitly list as a challenge for long sequences and resource-constrained devices. VGGT-SLAM [ 5 ] extends feed-forward VGGT to long sequences by running VGGT on overlapping keyframe windows to form submaps, then aligning them via 15-DOF homographies on the SL(4) manifold to address the projective ambiguity that arises with uncalibrated monocular, offline input. The pipeline augments sequential alignment with loop-closure constraints and solves a global factor-graph optimization on SL(4). Spatial Tracker V2 [ 6 ] unifies 3D point tracking, monocular video depth, and camera pose estimation in a feed-forward pipeline, achieving state-of-the-art results. The authors report up to 50\times faster runtime than dynamic 3D reconstruction approaches, however, memory usage grows with sequence length, and a \sim 150-frame input may require more than 40GB of GPU VRAM. 3 Methods 3.1 Global alignment The number of input frames that VGGT can process for 3D generation is constrained by GPU memory. To scale to longer sequences, we construct submaps with VGGT and then align them, thereby making efficient use of memory. We partition the input video stream into non-overlapping blocks of consecutive frames. To align submaps globally and efficiently, we keep a rolling global reference extrinsic matrix and exploit that VGGT expresses all poses within in a block of images in the coordinate system of the first frame of the block (see Fig. 2 for illustration). Figure 2 : The input stream is partitioned into blocks, with key frames serving as anchors for alignment. Let each