GGPT: Geometry-Grounded Point Transformer

Yutong Chen, Yiming Wang, Xucong Zhang, Sergey Prokudin, Siyu Tang, ETH Zurich; Delft University of Technology · 2026

Read GGPT: Geometry-Grounded Point Transformer by Yutong Chen, Yiming Wang, Xucong Zhang, Sergey Prokudin, Siyu Tang, ETH Zurich; Delft University of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11174.txt.

Introduction

Yutong Chen and coauthors present GGPT, a point transformer that grounds feed-forward sparse-view 3D reconstruction in geometry. It fixes flaws that come from predicting dense point maps straight from images. An improved structure-from-motion pipeline based on dense feature matching first estimates accurate camera poses and partial point clouds from the sparse views. GGPT then refines dense point maps under partial-geometry supervision, restoring fine detail and filling textureless gaps. The model also works across architectures and datasets.

Catalog record

CREATORYutong Chen, Yiming Wang, Xucong Zhang, Sergey Prokudin, Siyu Tang, ETH Zurich; Delft University of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE93075
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GGPT: Geometry-Grounded Point Transformer Yutong Chen, Yiming Wang, Xucong Zhang, Sergey Prokudin, Siyu Tang, ETH Zurich; Delft University of Technology Abstract Recent feed-forward networks have achieved remarkable progress in sparse-view 3D reconstruction by predicting dense point maps directly from RGB images. However, they often suffer from geometric inconsistencies and limited fine-grained accuracy due to the absence of explicit multi-view constraints. We introduce the G eometry- G rounded P oint T ransformer (GGPT), a framework that augments feed-forward reconstruction with reliable sparse geometric guidance. We first propose an improved Structure-from-Motion pipeline based on dense feature matching and lightweight geometric optimisation to efficiently estimate accurate camera poses and partial 3D point clouds from sparse input views. Building on this foundation, we propose a geometry-guided 3D point transformer that refines dense point maps under explicit partial-geometry supervision using an optimised guidance encoding. Extensive experiments demonstrate that our method provides a principled mechanism for integrating geometric priors with dense feed-forward predictions, producing reconstructions that are both geometrically consistent and spatially complete, recovering fine structures and filling gaps in textureless areas. Trained solely on ScanNet++ with VGGT predictions, GGPT generalises across architectures and datasets, substantially outperforming state-of-the-art feed-forward 3D reconstruction models in both in-domain and out-of-domain settings. Project website: https://chenyutongthu.github.io/research/ggpt . Figure 1 : Top row: the dense point maps predicted by feed-forward methods ( e.g . VGGT Wang et al. 2025b ) struggle with multi-view geometric consistency resulting in large error. Bottom row: with geometric guidance, our GGPT refines the dense point maps to enhance global alignment and 3D consistency, substantially reducing the reconstruction error. 1 Introduction In recent years, feed-forward 3D reconstruction networks have rapidly emerged, aiming to recover complete scene geometry directly from sparse RGB inputs. Starting with DUSt3R Wang et al. 2024b , which first demonstrated that a single transformer can jointly predict camera poses and dense 3D point maps from uncalibrated image pairs, subsequent works such as MASt3R Leroy et al. 2024 and VGGT Wang et al. 2025b have further advanced this paradigm. Trained on large-scale multi-view datasets, these vision transformers can now produce dense, visually coherent reconstructions in a single forward pass. As a result, feed-forward 3D reconstruction has become one of the most promising directions in 3D vision, offering fast, scalable, and unified reconstruction pipelines. Yet, despite their impressive visual results, closer inspection reveals that these models still struggle with multi-view geometric consistency (see Fig. 1 ). Their reconstructions often exhibit multi-layer artifacts and deviate from the ground truth, particularly when applied outside their training distribution. In practice, although these networks can predict plausible 3D structures from limited views, they often fail to recover geometry that is sufficiently accurate and consistent across viewpoints. This limitation becomes evident in out-of-domain scenarios, such as medical or surgical scenes, or human data, where feed-forward predictions can deviate substantially from the true underlying geometry. In contrast, Structure-from-Motion (SfM) Schönberger and Frahm 2016 remains firmly grounded in geometric principles Hartley and Zisserman 2003 , producing geometrically consistent reconstructions. However, SfM pipelines remain fragile under wide baselines, low overlap, or limited viewpoints, and typically recover only sparse geometry structure. This contrast highlights a clear opportunity: to combine the completeness and efficiency of feed-forward 3D reconstruction with the geometric accuracy and generalisation of SfM. Building on this motivation, several recent studies have explored similar ideas Guo et al. 2025 ; Zuo et al. 2025 ; Jang et al. 2025 ; Keetha et al. 2025 , showing that incorporating sparse geometric guidance can improve dense feed-forward reconstructions. However, existing approaches remain limited in two key aspects. First , they often rely on unrealistic SfM guidance, such as pseudo SfM points sampled from ground truth Jang et al. 2025 ; Zuo et al. 2025 ; Keetha et al. 2025 or SfM results obtained from densely captured video sequences Guo et al. 2025 , which are rarely available in real-world sparse-view scenarios. This dependency largely reflects the lack of robustness and efficiency of conventional SfM pipelines under limited input views. Second , prior methods refine predictions in 2D image space via depth-map diffusion Guo et al. 2025 or image transformers Jang et al. 2025 ; Keetha et al. 2025 , which inherently constrain the model to view-dependent reasoning. As a result, they fail to exploit the explicit 3D structure of the scene and cannot enforce true cross-view geometric consistency. In this work, we introduce GGPT ( G eometry- G rounded P oint T ransformer), a geometry-guided framework that refines dense feed-forward reconstructions using accurate geometric guidance obtained from an improved SfM pipeline, and does so directly in 3D space. First, we revisit SfM under limited input views and introduce an improved pipeline that integrates dense matchers Edstedt et al. 2024 ; Zhang et al. 2025b with a lightweight optimisation procedure. Concretely, we estimate camera poses from a compact set of high-confidence correspondences and then triangulate all valid matches using a direct linear transform. Compared with state-of-the-art SfM methods designed for unconstrained captures, including VGGT+BA Wang et al. 2024a ; Wang et al. 2025b and MASt3R-SfM Duisterhof et al. 2025 , our SfM pipeline achieves higher accuracy thanks to the rich geometric constraints provided by the dense correspondences, and also improved efficiency thanks to the separation of non-linear BA optimisation and linear triangulation. Its robustness and efficiency also enable the construction of realistic SfM supervision without requiring densely captured multi-view sequences. Second, we introduce a variant of a lightweight 3D Point Transformer Wu et al. 2024 that jointly processes dense point maps from feed-forward models and geometrically grounded partial point cloud from our SfM pipeline, predicting residual corrections for every point. By reasoning directly in a global 3D coordinate space, GGPT propagates the geometric accuracy of triangulated SfM points to dense but noisy feed-forward predictions. In contrast to prior depth completion methods that operate on 2D image tokens and channel-wise fuse geometry and image features Keetha et al. 2025 ; Jang et al. 2025 ; Zuo et al. 2025 ; Guo et al. 2025 , GGPT performs attention on the two point clouds directly in 3D, where spatial proximity, not pixel coordinates, defines receptive fields. This design explicitly enforces multi-view geometric consistency and produces globally aligned and metrically coherent dense reconstructions. We conduct extensive experiments to analyse both our improved SfM pipeline and Geometry-Grounded Point Transformer. Our SfM framework achieves better performance and efficiency than state-of-the-art SfM alternatives Wang et al. 2024a ; Duisterhof et al. 2025 , making it a practical and robust solution for applications beyond the scope of this work. When conditioned on the same SfM guidance, our proposed point transformer predicts significantly more accurate dense points than prior depth completion networks Guo et al. 2025 ; Zuo et al. 2025 ; Jang et al. 2025 ; Keetha et al. 2025 . Overall, thanks to its intuitive and modular design, our method can be seamlessly integrated with various feed-forward 3D reconstruction models at inference time, without requiring any fine-tuning. Remarkably, despite being trained only on ScanNet++ Yeshwanth et al. 2023 using VGGT Wang et al. 2025b dense predictions, GGPT generalises effectively across architectures and datasets, substantially improving the performance of state-of-the-art feed-forward models in both in-domain and out-of-domain settings. The strong generalisation results further suggest that

Read the complete text