Geometric Context Transformer for Streaming 3D Reconstruction

Lin-Zhuo Chen ∗ Jian Gao ∗ Yihang Chen Ka Leong Cheng Yipengjing Sun Liangxiao Hu Nan Xue Xing Zhu Yujun Shen Yao Yao † Yinghao Xu †‡ ∗ Equal Contribution † Corresponding Author ‡ Project Lead · 2026

Read Geometric Context Transformer for Streaming 3D Reconstruction by Lin-Zhuo Chen ∗ Jian Gao ∗ Yihang Chen Ka Leong Cheng Yipengjing Sun Liangxiao Hu Nan Xue Xing Zhu Yujun Shen Yao Yao † Yinghao Xu †‡ ∗ Equal Contribution † Corresponding Author ‡ Project Lead (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14141.txt.

Introduction

Chen and eleven colleagues present LingBot-Map, a feed-forward foundation model that rebuilds scenes from streaming video with geometric accuracy and temporal consistency. Its geometric context transformer joins three aids: an anchor context, a window tied to poses, and a memory of the path. Together they ground coordinates, gather dense cues, and correct long-range drift. The compact streaming state supports stable inference near 20 frames per second on 518 by 378 inputs over sequences beyond 10,000 frames, beating streaming and iterative rivals across benchmarks.

Catalog record

CREATORLin-Zhuo Chen ∗ Jian Gao ∗ Yihang Chen Ka Leong Cheng Yipengjing Sun Liangxiao Hu Nan Xue Xing Zhu Yujun Shen Yao Yao † Yinghao Xu †‡ ∗ Equal Contribution † Corresponding Author ‡ Project Lead
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85972
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geometric Context Transformer for Streaming 3D Reconstruction Lin-Zhuo Chen ∗ Jian Gao ∗ Yihang Chen Ka Leong Cheng Yipengjing Sun Liangxiao Hu Nan Xue Xing Zhu Yujun Shen Yao Yao † Yinghao Xu †‡ ∗ Equal Contribution † Corresponding Author ‡ Project Lead Abstract Streaming 3D reconstruction aims to recover 3D information, such as camera poses and point clouds, from a video stream, which necessitates geometric accuracy, temporal consistency, and computational efficiency. Motivated by the principles of Simultaneous Localization and Mapping (SLAM), we introduce LingBot-Map , a feed-forward 3D foundation model for reconstructing scenes from streaming data, built upon a geometric context transformer (GCT) architecture. A defining aspect of LingBot-Map lies in its carefully designed attention mechanism, which integrates an anchor context, a pose-reference window, and a trajectory memory to address coordinate grounding, dense geometric cues, and long-range drift correction, respectively. This design keeps the streaming state compact while retaining rich geometric context, enabling stable efficient inference at around 20 FPS on 518 × 378 518\times 378 resolution inputs over long sequences exceeding 10,000 frames. Extensive evaluations across a variety of benchmarks demonstrate that our approach achieves superior performance compared to both existing streaming and iterative optimization-based approaches. \metadata [Website] https://technology.robbyant.com/lingbot-map \metadata [Github] https://github.com/robbyant/lingbot-map \begin{overpic}[./radar.pdf] \end{overpic} Figure 1 : Comparison of LingBot-Map with State-of-the-Art streaming reconstruction methods. 1 Introduction \begin{overpic}[figures/teaser.pdf] \end{overpic} Figure 2 : Visualization results of LingBot-Map in VO mode on varying scenes: Given a continuous video stream, LingBot-Map performs efficient streaming 3D reconstruction with accurate camera pose estimation and high-quality point cloud reconstruction. Examples include Real-World Captured Aerial Video ( top-left ), World Model Generated Cartoon Video ( top-right ), Long-sequence Multi-Room Traversal Video ( middle ), and Long-sequence Outdoor Driving video ( bottom ). We perceive the world through a continuous stream of visual input, yet our spatial memory is not a faithful recording of every moment: it is sparse, structured, and efficient. Rather than retaining every observation, human spatial cognition selectively preserves only the most essential cues, enabling coherent navigation through large-scale environments over extended periods. Can we build machines that operate similarly, performing streaming 3D reconstruction from continuous visual input, selectively and efficiently? In recent years, 3D foundation models have advanced rapidly, with methods such as VGGT [ 75 ] and Depth Anything 3 [ 37 ] directly predicting camera poses, depth maps, and dense point maps from multiple images in a single feed-forward pass. However, these successes are largely confined to offline settings, where the full image set is available and processed globally. Recent efforts [ 79 , 103 , 28 , 33 ] have begun to adapt these capabilities to streaming reconstruction, but existing approaches still exhibit limited robustness to long sequences and complex scenes. The core difficulty is selective context management: balancing rich geometric context for long-term consistency with a compact state for efficient inference. Existing methods adopt different strategies to manage streaming context, each involving distinct trade-offs. CUT3R [ 79 ] maintains a persistent recurrent-style state, but its aggressive compression can lead to state forgetting and weak retention of essential geometric priors. In contrast, StreamVGGT [ 103 ] and Stream3R [ 28 ] adopt causal attention and caching, yet retain near-complete history without explicit selection, mixing useful geometry with redundancy and causing memory and computation to grow rapidly. A third line of work, such as VGGT-SLAM [ 42 ] and MASt3R-SLAM [ 46 ] , integrates learned 3D models with classical SLAM backends. While these methods incorporate structured context management through keyframe selection and pose-graph maintenance, such selection relies on hand-crafted heuristics rather than learned priors, and iterative optimization further limits real-time applicability. These observations point to a key design principle: the streaming state should selectively retain what matters most, not merely how much, and this selection should be grounded in geometric priors yet learned end-to-end from data. To this end, we introduce LingBot-Map , a streaming foundation model built around Geometric Context Attention (GCA) that realizes this principle within a unified attention framework. The design draws on a key insight from classical SLAM systems: robust real-time reconstruction requires maintaining distinct types of spatial context—a reference frame for coordinate grounding, a local window for dense local geometry estimation, and a global map for drift correction. Accordingly, GCA explicitly maintains three complementary types of context: an anchor context for coordinate and scale grounding, a local pose-reference window that retains dense visual features from recent frames for accurate local geometry estimation, and a trajectory memory that compresses the full observation history into compact per-frame tokens for global consistency. While the context structure is motivated by classical reconstruction principles, GCA replaces hand-crafted optimization with end-to-end learned attention that adaptively weights, encodes, and compresses information within each context type. This structured yet learned representation ensures stable and efficient inference even over arbitrarily long sequences, with nearly constant memory and computation per frame, as context beyond the local window is compressed into compact per-frame tokens. To scale up training effectively, we employ a progressive training strategy combined with context parallelism [ 20 ] , along with a relative loss formulation that facilitates stable optimization on long sequences. This allows LingBot-Map to learn efficiently from diverse, large-scale 3D datasets. We evaluate LingBot-Map on comprehensive benchmarks, including Oxford Spires, 7-Scenes, Tanks and Temples, ETH3D, and show consistent improvements over existing streaming methods in both camera pose estimation and dense 3D reconstruction quality. Our contributions are summarized as follows: • We introduce LingBot-Map , a streaming 3D foundation model built around Geometric Context Attention (GCA), which maintains three complementary context types – anchor, pose-reference window, and trajectory memory – for efficient and consistent long-sequence streaming inference. • We propose an efficient training recipe based on progressive training and context parallelism with a relative loss formulation for stable long-sequence optimization. • We demonstrate that LingBot-Map achieves state-of-the-art performance on multiple benchmarks (Oxford Spires, Tanks and Temples, ETH3D, and 7-Scenes), significantly outperforming existing streaming approaches in reconstruction quality and inference speed. 2 Related Work Traditional 3D Reconstruction. Traditional 3D reconstruction methods mainly include Structure-from-Motion (SfM) [ 62 , 56 , 48 ] , Simultaneous Localization and Mapping (SLAM) [ 44 , 45 , 4 ] , and Multi-View Stereo (MVS) [ 15 , 57 , 91 ] . SfM and SLAM recover camera poses and scene geometry from multi-view observations, where SfM typically operates offline on unordered image collections, while SLAM processes video streams online. These systems are usually complex and highly modular, typically centered around optimization-based bundle adjustment for camera pose estimation. In contrast, MVS focuses on dense reconstruction given known camera poses. Over the past decade, many works have explored replacing individual components of these pipelines with deep learning modules, particularly for feature extraction [ 12 ] and matching [ 54 , 65 ] . More recently, several approaches attempt to implement SfM, SLAM, or MVS in an end-to-end manner, such as VGGTSfM [ 76 ] , DROID-SLAM [ 69 ] , and MVSNet [ 91 , 92 ] . 3D Foundation Model. DUSt3R [ 81 ] represents a paradigm shift in feed-forward 3D reconstruction. Given a collection of unposed images, DUSt3R directly regresses a dense 3D reconstruction of the scene without explicit geometric modeling. However, DUSt3R [ 81

Read the complete text