Déjà View: Looping Transformers for Multi-View 3D Reconstruction

Alessandro Burzio* Tobias Fischer* Sven Elflein Qunjie Zhou, Riccardo de Lutio Jiawei Ren Jiahui Huang Shengyu Huang, Marc Pollefeys Laura Leal-Taixé Zan Gojcic Haithem Turki · 2026

Read Déjà View: Looping Transformers for Multi-View 3D Reconstruction by Alessandro Burzio* Tobias Fischer* Sven Elflein Qunjie Zhou, Riccardo de Lutio Jiawei Ren Jiahui Huang Shengyu Huang, Marc Pollefeys Laura Leal-Taixé Zan Gojcic Haithem Turki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30215.txt.

Introduction

Alessandro Burzio and eleven co-authors present DéjàView, a looped transformer for multi-view reconstruction. Feed-forward transformers have grown past a billion parameters, yet layers often repeat operations. DéjàView applies one looped block recurrently to per-view features. Trained once, it matches or beats larger baselines across five benchmarks. It uses a fraction of their parameters at comparable compute, with step count as a knob. The looped design beats an identical variant with independent parameters, pointing to explicit iteration as a stronger inductive bias.

Catalog record

CREATORAlessandro Burzio* Tobias Fischer* Sven Elflein Qunjie Zhou, Riccardo de Lutio Jiawei Ren Jiahui Huang Shengyu Huang, Marc Pollefeys Laura Leal-Taixé Zan Gojcic Haithem Turki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46284
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30215

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Déjà View: Looping Transformers for Multi-View 3D Reconstruction Alessandro Burzio* Tobias Fischer* Sven Elflein Qunjie Zhou, Riccardo de Lutio Jiawei Ren Jiahui Huang Shengyu Huang, Marc Pollefeys Laura Leal-Taixé Zan Gojcic Haithem Turki Abstract Recent feed-forward 3D reconstruction transformers have scaled to over a billion parameters, following the broader trend of increasing model capacity in computer vision. Yet emerging evidence suggests that contiguous transformer layers often behave like repeated applications of similar operations ( 27 ) , and multi-view reconstruction transformers refine their predictions progressively across decoder depth ( 56 ) . We posit that model depth partially buys iteration, paid for inefficiently in unique parameters, and instead make that iteration explicit in architecture. Our model, DéjàView, applies a single looped transformer block recurrently to per-view features for K K refinement steps. Trained once, it exposes K K as an inference-time compute knob, matching or outperforming substantially larger feed-forward baselines across five reconstruction benchmarks spanning indoor, outdoor, object-centric, and driving scenes, while using a fraction of their parameters and comparable or lower compute. Importantly, the same looped block formulation outperforms an otherwise identical variant with independent per-step parameters under matched training data and compute, suggesting that explicit iteration is not merely a compute-efficient substitute for capacity but a stronger inductive bias for multi-view 3D reconstruction. We release code and model on our project page https://research.nvidia.com/labs/dvl/projects/dvlt/ . Figure 1 : DéjàView. Given multiple input views (top-left) , DéjàView reconstructs camera poses and consistent depth by repeatedly applying the same transformer block, with the number of refinement steps K exposed as an inference-time compute knob. Decoding the intermediate state of a single K{=}16 forward pass at iterations k\in\{2,4,8,16\} shows progressively sharper geometry and more accurate camera poses ( right ; frustums are colored by per-camera error). Across five benchmarks (bottom-left) , DéjàView matches or surpasses much larger feed-forward baselines at a small fraction of their parameter count (dot area). 1 Introduction Recovering 3D structure from images has traditionally relied on a Structure-from-Motion (SfM) pipeline ( 52 ; 46 ) . These systems decompose reconstruction into feature extraction and matching, pose estimation, triangulation, and global bundle adjustment, yielding an inherently iterative process that alternates between registering new views and re-adjusting previous estimates. More recently, feed-forward methods ( 71 ; 35 ; 67 ; 38 ; 73 ; 31 ; 68 ) have replaced this pipeline with an end-to-end transformer that regresses 3D geometry from images in a single forward pass. Their gains, in line with the broader trend in computer vision ( 14 ; 86 ; 48 ; 45 ; 32 ) , have largely been driven by scaling model capacity through backbone depth and width ( 38 ) . Yet iteration may not have disappeared at all — it may simply have been absorbed into network depth. 27 show that for some applications the L layers of a Vision Transformer (ViT) can be replaced with K\ll L recurrent applications of a looped block with little loss in accuracy. 56 further probe the decoder of DUSt3R ( 71 ) layer-by-layer and find that its pointmap predictions are themselves iteratively refined across depth, despite the layers being independently parameterized. Together, these observations suggest that part of the benefit of depth in modern reconstruction transformers arises from implicit iterative refinement, at the cost of redundant layer-specific parameters. We therefore make this iterative process explicit in the architecture, rather than relying on model depth to realize it implicitly. Starting from per-view DINOv2 ( 45 ) features, we apply a single looped transformer block recurrently for K refinement steps. By sampling K from [K_{\text{min}},K_{\text{max}}] during training, a single checkpoint exposes K as an inference-time compute knob without retraining. Analyzing the trained recurrence reveals that it does not converge to a fixed point. Instead, each step progressively aligns the state’s direction toward its endpoint, a regime we call directional refinement . The resulting model, DéjàView, matches or surpasses much larger feed-forward baselines across five reconstruction benchmarks spanning indoor, outdoor, object-centric, and driving scenes, at a small fraction of their parameters and comparable or lower compute. Importantly, we show that our looping formulation with shared weights significantly outperforms an otherwise identical variant with independent per-step parameters under the same training data and compute budget. We take this as evidence that iterative refinement with shared weights is a viable alternative to parameter scaling for 3D reconstruction. We summarize our contributions as follows: • DéjàView, a looping transformer for multi-view 3D reconstruction that applies a single shared block recurrently to a DINOv2-initialized state, with each step conditioned on a continuous time interval. • A variable- K training recipe in which the step count is sampled per batch from [K_{\text{min}},K_{\text{max}}] , yielding a single checkpoint that exposes compute as an inference-time knob. • State-of-the-art reconstruction quality across five challenging benchmarks, at a small fraction of the parameters and comparable or lower compute. 2 Related Work Our work draws on several lines of research spanning 3D reconstruction, iterative refinement for geometry estimation, and weight-tied network design. Multi-view 3D reconstruction. Classical Structure-from-Motion ( 52 ; 46 ) recovers geometry through feature matching, pose estimation, and bundle adjustment, but is brittle on in-the-wild scenes with weak texture or dynamic content. Learning-based methods have progressively replaced individual stages of this pipeline, from multi-view stereo ( 81 ) to fully end-to-end systems. DUSt3R ( 71 ) and MASt3R ( 35 ) regress pairwise pointmaps from a CroCo-pretrained ( 75 ) backbone, while VGGT ( 67 ) and concurrent methods ( 73 ; 38 ; 31 ; 68 ) process all views jointly through a DINOv2 ( 45 ) -based transformer, with extensions to incremental capture ( 66 ; 74 ) , large view counts ( 79 ; 16 ) , pose-free Gaussian splatting ( 83 ) , multiple dense geometric quantities ( 17 ) and dynamic scenes ( 87 ; 58 ; 43 ) . All of these systems use a fixed-depth architecture with many unique parameters. We instead frame multi-view reconstruction as iterative refinement, matching the quality of these deeper feed-forward networks at a fraction of the parameter count while exposing the number of refinement steps as an inference-time compute knob. Iterative refinement. RAFT ( 60 ) introduced GRU-based iterative refinement of a per-pixel flow field, an idiom since broadly applied to stereo matching ( 40 ) , visual SLAM ( 61 ; 24 ) , and multi-view stereo ( 65 ) : in each case, a lightweight recurrent updater iteratively refines geometry on top of a fixed feature backbone. iLRM ( 29 ) extends iterative refinement to feed-forward 3D Gaussian splatting by treating successive (unshared) transformer layers as optimization steps over a scene representation decoupled from the input views. Unlike RAFT-style designs, where a lightweight recurrent updater iterates on a precomputed correspondence volume from a fixed feature backbone, our recurrence interleaves cross-view reasoning and refinement: each step is a full transformer block with frame and global attention, applied to per-view tokens initialized from a DINOv2 patch encoder. Cross-view reasoning and refinement therefore occur within the same repeated computation, rather than being assigned to separate stages. Unlike iLRM, the block is shared across all K steps and refines the per-view tokens themselves rather than a decoupled

Read the complete text