Keyframe-Based Feed-Forward Visual Odometry
Weichen Dai Wenhan Su, Da Kong, Yuhang Ming, Wanzeng Kong · 2026
Read Keyframe-Based Feed-Forward Visual Odometry by Weichen Dai Wenhan Su, Da Kong, Yuhang Ming, Wanzeng Kong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16020.txt.
Introduction
Weichen Dai, Wenhan Su, Da Kong, Yuhang Ming, and Wanzeng Kong streamline visual odometry in a 2026 arXiv paper. Visual odometry estimates motion from camera images. SLAM adds mapping on top of it. Visual foundation models have reshaped both fields. A single feed-forward network can now estimate pose and dense structure. Older pipelines leaned on keyframes: select frames that carry the most value. Newer model-based methods often skip that wisdom.
They process raw image sequences indiscriminately instead. That wastes compute on redundant frames. The authors bring keyframes back into a feed-forward design, per the title. Picking key frames well boosts both efficiency and accuracy. Robots and AR devices run on tight power budgets. Odometry that does more with fewer frames fits their limits.
Catalog record
| CREATOR | Weichen Dai Wenhan Su, Da Kong, Yuhang Ming, Wanzeng Kong |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16020
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Keyframe-Based Feed-Forward Visual Odometry Weichen Dai Wenhan Su, Da Kong, Yuhang Ming, Wanzeng Kong Abstract The emergence of visual foundation models has revolutionized visual odometry (VO) and SLAM, enabling pose estimation and dense reconstruction within a single feed-forward network. However, unlike traditional pipelines that leverage keyframe methods to enhance efficiency and accuracy, current foundation model based methods, such as VGGT-Long, typically process raw image sequences indiscriminately. This leads to computational redundancy and degraded performance caused by low inter-frame parallax, which provides limited contextual stereo information. Integrating traditional geometric heuristics into these methods is non-trivial, as their performance depends on high-dimensional latent representations rather than explicit geometric metrics. To bridge this gap, we propose a novel keyframe-based feed-forward VO. Instead of relying on hand-crafted rules, our approach employs reinforcement learning to derive an adaptive keyframe policy in a data-driven manner, aligning selection with the intrinsic characteristics of the underlying foundation model. We train our agent on TartanAir dataset and conduct extensive evaluations across several real-world datasets. Experimental results demonstrate that the proposed method achieves consistent and substantial improvements over state-of-the-art feed-forward VO methods. I INTRODUCTION Visual Odometry (VO) and Simultaneous Localization and Mapping (SLAM) constitute the backbone for robotics, enabling autonomous systems to estimate their pose and reconstruct maps of unknown environments using visual input. Traditional VO and SLAM approaches [ 1 , 2 , 3 ] , rely on multi-stage pipelines [ 4 ] that involve feature extraction, tracking and nonlinear optimization under explicit geometric constraints [ 5 ] . However, these methods suffer from inherent limitations: they require extensive expert knowledge to carefully tune parameters and design heuristics in order to handle a wide range of corner cases encountered in real-world scenarios, including dynamic scenes [ 6 ] , low-texture environments [ 7 ] , and severe illumination changes [ 8 ] . The recent emergence of feed-forward visual foundation models has made it possible to establish a unified VO pipeline that requires no calibration and tuning with great generalization capabilities. Pioneering works such as DUSt3R [ 9 ] and its successor MASt3R [ 10 ] have demonstrated the feasibility of jointly estimating camera parameters and dense 3D geometry from uncalibrated image pairs. Subsequent models, including CUT3R [ 11 ] and Fast3R [ 12 ] , have further optimized this paradigm. Most recently, VGGT [ 13 ] achieved state-of-the-art reconstruction quality, generating stable and accurate local 3D maps directly from raw monocular RGB input. Extensions such as VGGT-SLAM [ 14 ] and VGGT-Long [ 15 ] have introduced chunking and loop closure to achieve globally aligned long-term pose estimation. Despite these advances, these methods generally lack filtering for input frames. They do not mitigate the negative impact of low-quality and redundant frames on local estimation and do not address the computational and memory reduction caused by similar input frames. Extensive research in both VO/SLAM [ 16 ] and Structure-from-Motion (SfM) [ 17 ] has demonstrated that a well-designed keyframe method can effectively improve performance while reducing computational burden. However, translating traditional keyframe method to foundation models is not easy. Conventional methods rely on explicit geometric metrics, whereas foundation models like VGGT operate as ”black boxes” with complex, high-dimensional latent spaces. Designing a rule-based keyframe method that aligns with the internal characteristics of these models, without explicit knowledge of their internal mechanics, remains a significant challenge. To this end, we propose a keyframe-based feed-forward VO method that using VGGT as the backbone with a reinforcement learning–based keyframe method. The keyframe method adaptively learns, in a data-driven manner, to align with the intrinsic characteristics of VGGT. This enables more efficient management of the sliding-window inputs and leads to improved overall localization accuracy. We train the proposed method on the synthetic TartanAir dataset [ 18 ] and conduct extensive experiments on several widely used real-world datasets, including EuRoC [ 19 ] , TUM-RGBD [ 20 ] , and KITTI [ 21 ] . The results consistently demonstrate significant performance improvements. Our main contributions can be summarized as follows: • We introduced a keyframe-based feed-forward VO method that leveraging the ability of VGGT with a data-driven RL based keyframe method. By aligning input frame selection with the characteristics of VGGT. • We introduced an adaptive keyframe method into 3D vision foundation models like VGGT. By aligning input frame selection with the intrinsic representation characteristics of foundation models, the proposed approach significantly improves localization performance. • The experiment results demonstrate strong generalization capability of the proposed method cross real-world datasets. II RELATED WORK II-A The State of VO In the field of VO, two primary technical approaches dominate: classic methods and end-to-end learning methods. Classic methods have demonstrated exceptional accuracy and stability in industrial-grade applications. The tracking methods are categorized into three typical paradigms: direct visual odometry, feature point matching, and hybrid semi-direct methods. Direct methods, exemplified by DSO [ 1 ] , use pixel intensity to identify key image regions and analyze their motion across frames to estimate camera pose. Feature point methods like ORB-SLAM3 [ 2 ] , extract feature points from images and track their positional changes to estimate motion. Semi-direct methods such as SVO [ 22 ] combine feature point detection with brightness analysis of surrounding image patches to estimate motion. The recent paradigm shift driven by deep learning has given rise to data-driven, end-to-end methods [ 23 ] [ 24 ] . In representative work, DPVO [ 3 ] employs differentiable optimization layers and recurrent network architectures to realize a motion estimation system based on image patch tracking. LEAP-VO [ 25 ] employs long-term point tracking with anchor-based dynamic estimation and temporal probability modeling to build a robust visual odometry system for dynamic scenes. MAC-VO [ 26 ] introduces a stereo visual odometry framework that leverages a metrics-aware covariance model for keypoint selection and residual weighting in pose graph optimization. II-B 3D Vision Foundation Models In recent, end-to-end foundation models have emerged as one of the most prominent research frontiers in the field of 3D computer vision. Primarily built upon the transformer architecture [ 27 ] , these models aim to achieve dense geometric reconstruction directly from images without requiring prior knowledge of camera poses or calibration parameters. Pioneering research, represented by DUSt3R [ 9 ] and its successor MASt3R [ 10 ] , first demonstrated the feasibility of jointly estimating camera parameters and dense 3D structures from uncalibrated image pairs. Subsequently, models such as CUT3R [ 11 ] and Fast3R [ 12 ] have further refined this technical paradigm, achieving a superior balance between computational efficiency and reconstruction accuracy. Currently, the state-of-the-art model VGGT [ 13 ] is capable of generating highly stable and precise local 3D maps directly from raw RGB inputs. Concurrently, new novel models like {\pi}^{3} [ 28 ] , Depth Anything 3 [ 29 ] , and MapAnything [ 30 ] are continue to drive rapid advancements in this domain. However, a significant limitation shared by these models is their substantial computational and memory overhead, which severely restricts their practical deployment in tasks such as long-sequence visual localization or large-scale scene reconstruction. To address this challenge, recent works including VGGT-SLAM [ 14 ] , VGGT-Long [ 15 ] , and FlashVGGT [ 31 ]