SLAM-Former: Putting SLAM into One Transformer
Yijun Yuan, Zhuoguang Chen, Kenan Li, Weibang Wang, Hang Zhao · 2025
Read SLAM-Former: Putting SLAM into One Transformer by Yijun Yuan, Zhuoguang Chen, Kenan Li, Weibang Wang, Hang Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16909.txt.
Introduction
Yijun Yuan and co-authors put robot mapping and tracking into one transformer in a 2025 paper. SLAM lets a robot map a place while tracking its own position in it. Classic systems split this into a frontend and a backend. Their model, SLAM-Former, houses both parts in a single transformer that runs them in turns.
Each part boosts the other as they alternate. Tests show scores that match or beat top dense SLAM methods. A single learned system could simplify robot stacks that now mix many hand-built parts. Simpler stacks are easier to train, ship, and debug across varied robots and scenes.
Catalog record
| CREATOR | Yijun Yuan, Zhuoguang Chen, Kenan Li, Weibang Wang, Hang Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16909
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SLAM-Former: Putting SLAM into One Transformer Yijun Yuan, Zhuoguang Chen, Kenan Li, Weibang Wang, Hang Zhao Abstract We present SLAM-Former, a novel neural approach that integrates full SLAM capabilities into a single transformer. Similar to traditional SLAM systems, SLAM-Former comprises both a frontend and a backend that operate in tandem. The frontend processes sequential monocular images in real-time for incremental mapping and tracking, while the backend performs global refinement to ensure a geometrically consistent result. This alternating execution allows the frontend and backend to mutually promote one another, enhancing overall system performance. Comprehensive experimental results demonstrate that SLAM-Former achieves superior or highly competitive performance compared to state-of-the-art dense SLAM methods. Figure 1 : SLAM-Former is a unified Transformer for SLAM. Traditional SLAM employs a multi-model pipeline for frontend and backend tasks. In contrast, SLAM-Former integrates the full SLAM functionality within one transformer, achieving consistent poses and dense maps. 1 Introduction In the field of robotic perception, Simultaneous Localization and Mapping (SLAM) plays a great significance. It allows robots to construct a map of an unknown environment while simultaneously tracking their own location. This capability is essential for robots to navigate autonomously and carry out tasks in a variety of environments. Early SLAM algorithms primarily focused on localization and mapping using sparse points, such as ORB-SLAM Mur_Artal_2015 and LSD-SLAM engel2014lsd . These methods are efficient and robust, but they may not offer detailed information about the surroundings. In contrast, dense mapping techniques aim to create a more detailed and continuous representation of the environment, mainly relying on LiDAR and RGB-D elseberg2013one ; 6162880 . With the rapid advancement of optical flow and multiview depth estimation techniques, recent research has achieved high-quality dense monocular SLAM using only images as input NEURIPS2021_89fcd07f ; yuan2025scenefactoryworkflowcentricunifiedframework ; murai2025mast3rslamrealtimedenseslam ; maggio2025vggtslamdensergbslam . These approaches leverage the capability of neural networks and computer vision algorithms to estimate depth and motion from a single camera, thereby creating dense maps without the need for additional sensors. Especially noteworthy is the trend of utilizing geometric foundation models such as DUSt3R wang2024dust3rgeometric3dvision and VGGT wang2025vggtvisualgeometrygrounded . These models reveal the high potential of data-driven 3D structure prediction. Their streaming variants, StreamVGGT zhuo2025streaming4dvisualgeometry and Stream3R lan2025stream3rscalablesequential3d , by carefully leveraging the attention key-value cache (KV cache), enable the model to process incremental visual inputs. We observe that SLAM methods using geometric foundation models as their reconstruction module, such as MASt3R-SLAM and VGGT-SLAM suffer from global inconsistency, since they rely on the alignment of local submaps. On the other hand, streaming methods such as StreamVGGT and Stream3R process incremental inputs without remapping the past, which can cause a significant mismatch between past and newly incoming data. In this work, we introduce a visual SLAM framework implemented within a single unified transformer architecture, named SLAM-Former. SLAM-Former consists of a frontend and a backend in the same transformer, working in cooperation, as depicted in Fig. 1 (b). The frontend operates in real-time on the sequential RGB images for keyframe selection and incremental map and pose updates. With the incremental output from the frontend, our backend periodically refines the map and poses globally at a lower frequency. The frontend and the backend promote each other in this alternating process. After each backend run, the transformer’s KV cache is updated to the frontend for further incremental operation. In return, the frontend provides initial results and the sequential order, assisting the backend in refinement. To empower a single transformer with all SLAM capabilities, we propose three training modes for SLAM-Former. Compared to the traditional SLAM pipeline which requires an additional loop detection module to close its pose graph, SLAM-Former’s backend accomplishes this with full attention and is equivalent to processing loop detection on a dense factor graph. SLAM-Former achieves significantly better reconstruction and state-of-the-art tracking performance on widely used Dense Mono SLAM benchmarks, compared to both calibrated and uncalibrated state-of-the-art methods. 2 Related Works 2.1 Dense RGB SLAM In recent years, research on dense SLAM using monocular cameras has achieved significant progress NEURIPS2021_89fcd07f ; zhu2023nicerslamneuralimplicitscene ; yuan2025scenefactoryworkflowcentricunifiedframework ; murai2025mast3rslamrealtimedenseslam ; maggio2025vggtslamdensergbslam , thanks to the application of deep learning techniques. Due to the absence of depth sensors, dense RGB SLAM requires optimizing the entire sequence of geometry and camera as a whole. Early works focus on reducing the computational cost of depth estimation. For instance, CodeSLAM bloesch2019codeslamlearningcompact and DeepFactors Czarnowski_2020 optimize the depth latent as an alternative. Borrowing the strength of MVSNet yao2018mvsnetdepthinferenceunstructured , Tandem relies on an external model, but it breaks the co-optimization structure koestler2022tandem . Conversely, DROID-SLAM NEURIPS2021_89fcd07f and SceneFactory yuan2025scenefactoryworkflowcentricunifiedframework incorporate deep optical flow model into the pipeline and co-optimize both with a speed-dense bundle adjustment. On the other hand, NeRF mildenhall2020nerfrepresentingscenesneural and Gaussian Splatting kerbl20233dgaussiansplattingrealtime based methods have emerged as a trend to reshape Dense SLAM. NeRF-SLAM methods rosinol2022nerfslamrealtimedensemonocular ; zhu2023nicerslamneuralimplicitscene and GS-SLAM methods yan2024gsslamdensevisualslam optimize the scene as a whole for a highly realistic novel view synthesis objective. Nonetheless, those rendering-based SLAMs are usually time-consuming, not reconstruction-affordable, and highly sensitive to blur and noise, which substantially restricts their use in real life. With the emergence of recent foundational geometry techniques, such as DUSt3R wang2024dust3rgeometric3dvision and VGGT wang2025vggtvisualgeometrygrounded , researchers have found new inspiration. MASt3R-SLAM murai2025mast3rslamrealtimedenseslam utilizes the advanced pairwise model MASt3R leroy2024groundingimagematching3d for high-quality calibration-free matching and geometry construction, demonstrating state-of-the-art performance under a traditional SLAM pipeline. On the other hand, VGGT-SLAM maggio2025vggtslamdensergbslam feeds submaps into VGGT and connects them using a novel SL(4) manifold, modeling the geometry distortion in foundational geometry for the first time. However, these methods rely on pair- or submap-wise optimization of geometry, which often causes conflict structure among frames. MASt3R-SLAM attempts to address this issue through TSDF-fusion 6162880 , which could only fix small mismatches. Meanwhile, VGGT-SLAM cannot deal with this issue because it only has connections of submaps in the front- and end-nodes. This inspires us to develop a frontend-backend SLAM structure that can properly and neatly address this issue. Figure 2 : The working pipeline of SLAM-Former. The frontend detects keyframes and performs incremental pose and map updates, while the backend performs global pose and map updates. The shared map token memory and the KV cache update mechanism ensure that the frontend and the backend promote each other, and this process is marked by blue arrows . 2.2 Feed-forward 3D Reconstruction Recently, DUSt3R wang2024dust3rgeometric3dvision has led a trend to regress the 3D structure with scalable training data directly. However, with processing image pairs, DUSt3R requires a global optimization for larger scenes, which lowers the inference efficiency. Several works have been proposed to address this limitation. Fast3R yang2025fast3r3dreconstruction1000 , VGGT wang2025vggtvisualgeometrygrounded , and Pi3 wang2025pi3scalablepermutationequivariantvisual process multi-view images in a single forward pass, avoiding time-consuming post-processing global optimization. All three are Transformer-based models for multi-view pointmap estimation. Fast3R highlights the ability to efficiently handle thousands of images, while VGGT demonstrates that a simple architecture with 3D multi-task learning and scalable training data can achieve state-of-the-art results. Pi3 further introduces a permutation-equivariant design that removes the dependence on a fixed reference view, enhancing robustness to input ordering and scalability. Besides feed-forward multi-view methods, recent feed-forward streaming approaches tackle 3D reconstruction online. Spann3R wang20243dreconstructionspatialmemory extends Dust3R to streaming by maintaining and interacting with a spatial memory. CUT3R wang2025continuous3dperceptionmodel