S-VGGT: Structure-Aware Subscene Decomposition for Scalable 3D Foundation Models Corresponding author: Wentao Cheng
Xinze Li 1 , Pengxu Chen 2 1 , Yiyuan Wang 3 1 , Weifeng Su 1 4 , Wentao Cheng 1 · 2026
Read S-VGGT: Structure-Aware Subscene Decomposition for Scalable 3D Foundation Models Corresponding author: Wentao Cheng by Xinze Li 1 , Pengxu Chen 2 1 , Yiyuan Wang 3 1 , Weifeng Su 1 4 , Wentao Cheng 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17625.txt.
Introduction
Li and co-authors present S-VGGT, a way to scale feed-forward 3D foundation models. Global attention costs grow quadratically with input length, which caps scene size. Token merging saves little and adds search overhead instead. S-VGGT splits frames into a few subscenes through a dense scene graph. Each part shares one reference frame, so parts run in parallel without alignment steps. The trick stacks with token-level speedups, they report.
Catalog record
| CREATOR | Xinze Li 1 , Pengxu Chen 2 1 , Yiyuan Wang 3 1 , Weifeng Su 1 4 , Wentao Cheng 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26433 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.17625
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
S-VGGT: Structure-Aware Subscene Decomposition for Scalable 3D Foundation Models Corresponding author: Wentao Cheng Xinze Li 1 , Pengxu Chen 2 1 , Yiyuan Wang 3 1 , Weifeng Su 1 4 , Wentao Cheng 1 Abstract Feed-forward 3D foundation models face a key challenge: the quadratic computational cost introduced by global attention, which severely limits scalability as input length increases. Concurrent acceleration methods, such as token merging, operate at the token level. While they offer local savings, the required nearest-neighbor searches introduce undesirable overhead. Consequently, these techniques fail to tackle the fundamental issue of structural redundancy dominant in dense capture data. In this work, we introduce S-VGGT , a novel approach that addresses redundancy at the structural frame level, drastically shifting the optimization focus. We first leverage the initial features to build a dense scene graph, which characterizes structural scene redundancy and guides the subsequent scene partitioning. Using this graph, we softly assign frames to a small number of subscenes, guaranteeing balanced groups and smooth geometric transitions. The core innovation lies in designing the subscenes to share a common reference frame, establishing a parallel geometric bridge that enables independent and highly efficient processing without explicit geometric alignment. This structural reorganization provides strong intrinsic acceleration by cutting the global attention cost at its source. Crucially, S-VGGT is entirely orthogonal to token-level acceleration methods, allowing the two to be seamlessly combined for compounded speedups without compromising reconstruction fidelity. Code is available at Github . I Introduction The advent of 3D foundation models has marked a new phase in multi-view geometry, enabling efficient, optimization-free reconstructions with strong generalization. Early pointmap prediction architectures such as DUSt3R [ 20 ] established the foundations for robust and unconstrained feed-forward 3D reconstruction. This progress was further advanced by memory-augmented approaches with persistent or spatial state [ 19 , 17 ] , as well as large-scale reconstruction systems capable of handling thousands of views [ 3 , 22 ] . End-to-end feed-forward Structure-from-Motion (SfM) pipelines [ 7 , 6 ] further showed that effective pose and depth estimation is possible with minimal reliance on bundle adjustment. Building on these advancements, global attention systems such as VGGT [ 18 ] and other large-scale models [ 8 , 21 ] have enabled efficient 3D reconstruction by recovering camera poses, depth maps, and point clouds in a single forward pass. However, a critical bottleneck remains, as the quadratic computational cost incurred by global attention grows rapidly with the input sequence length. This problem is particularly pronounced in dense capture scenarios, where frames exhibit significant visual redundancy, and the information gain from additional frames diminishes. In such cases, the challenge shifts to how we can efficiently extract the geometric details from each frame, ensuring that crucial spatial context is preserved without incurring prohibitive computational costs. Fig. 1: Comparison of VGGT (2.69 FPS) and S-VGGT (10.13 FPS) on a 500-frame scene. S-VGGT achieves a significant speedup by processing subscenes in parallel while maintaining reconstruction quality. A prominent line of work focuses on reducing redundancy at the token level. Techniques initially proposed for visual encoders [ 2 , 4 ] compress local representations by estimating token similarity and merging those judged redundant. Adaptations for 3D geometric models [ 14 ] follow the same principle and have demonstrated meaningful speedups. However, large-scale merging inherently requires computing similarity over a substantial token set, adding nontrivial overhead that partially offsets the acceleration. Furthermore, the fusion of multiple tokens modifies the underlying feature distribution, leading to a representation shift that may degrade geometric precision under high compression ratios. Our work is motivated by the observation that redundancy in dense sequences manifests primarily at the frame level. In scenarios where neighboring images share significant geometric overlap, the marginal information gain provided by global attention diminishes rapidly, making the quadratic computational cost unjustifiable. This phenomenon mirrors concepts in classical SfM, where strategies such as view graph sparsification [ 16 , 9 , 23 , 13 ] and reconstruction via submaps [ 11 , 12 ] utilize structural decomposition to tractable subproblems. Our approach capitalizes on this insight: by addressing redundancy at the frame level, we can reduce the effective input sequence length before attention is applied, thereby avoiding the quadratic cost at its source. Crucially, the successful mechanisms used in classical SfM, such as iterative pose refinement and explicit alignment, conflict with the forward only, single pass requirement of modern foundation models. As a result, we require a novel formulation that bridges the gap between structural efficiency and the single pass nature of geometric foundation models. To realize this vision, we introduce S-VGGT, a framework designed for efficient inference by targeting systematic redundancy. Our core strategy is to decompose the input sequence into a small number of coherent partitions that preserve the original spatial structure while reducing the effective frame count. We leverage the model’s intrinsic intermediate features to derive a density-aware affinity score, which captures the inter-frame correlations and guides the sequence decomposition. A key aspect of our approach is assigning a common reference frame to all subscenes. This mechanism allows for independent and parallel processing within a unified coordinate system, eliminating the need for explicit alignment after inference. As a result, we achieve significant reductions in global attention complexity, delivering substantial speedups without sacrificing reconstruction quality. We evaluate S-VGGT on several datasets, including ScanNet [ 5 ] , NRGBD [ 1 ] , and 7Scenes [ 15 ] , and demonstrate a significant speed advantage over VGGT. As shown in Fig. 1 , S-VGGT achieves substantial acceleration without compromising reconstruction fidelity. Importantly, our sequence optimization is fully orthogonal to token techniques, such as those used in FastVGGT [ 14 ] . This orthogonality enables seamless integration with existing token-based acceleration methods. By combining S-VGGT with techniques like token merging, we achieve even greater speedups, leveraging the strengths of both approaches to reduce computational bottlenecks while preserving reconstruction fidelity. II Method In this section, we introduce the key components of our method. First, we provide an overview of the foundational components of VGGT [ 18 ] , upon which our approach is built. We then detail our strategy for addressing frame-level redundancy through soft partitioning, followed by an explanation of how we evaluate frame similarity and organize frames into coherent subscenes. Next, we describe how the subscenes are processed in parallel and how we ensure consistency across them using anchor frame sharing. Finally, we present the optimization techniques used to achieve efficient grouping and show how they enable significant computational savings without compromising reconstruction quality. The overall framework of our method is illustrated in Fig. 2 . II-A Preliminaries We briefly summarize the relevant components of VGGT [ 18 ] that our method builds upon. Given a sequence of N RGB images \mathbf{I}=\{\mathbf{I}_{s}\}_{s=1}^{N} with each \mathbf{I}_{s}\in\mathbb{R}^{3\times H\times W} , VGGT first encodes each frame using a DINOv2 [ 10 ] backbone: \mathbf{F}_{s}=f_{\mathrm{DINO}}(\mathbf{I}_{s})\in\mathbb{R}^{P\times C}, producing P patch tokens per frame. Following the original architecture, one camera token and four register tokens are appended, so each frame contains (P+5) tokens in total. VGGT employs an alternating-attention backbone consisting of L layers. Each layer first performs frame-wise self-attention independently on all frames, and then applies a global cross-frame attention on the concatenation