Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself

Yuhang Dai, Xingyi Yang · 2026

Read Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself by Yuhang Dai, Xingyi Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14048.txt.

Introduction

Dai and Yang let feed-forward 3D reconstruction models improve themselves at test time without ground truth. Free Geometry masks some frames to build a self-supervised task, enforcing cross-view feature consistency between full and partial observations while preserving relations implied by held-out frames. Lightweight LoRA updates recalibrate a model in under 2 minutes per dataset, lifting camera pose accuracy by 3.73 percent and point-map prediction by nearly 3 percent.

Catalog record

CREATORYuhang Dai, Xingyi Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59408
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14048

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Free Geometry: Refining 3D Reconstruction from Longer Versions of Itself Yuhang Dai, Xingyi Yang Abstract Feed-forward 3D reconstruction models are efficient but rigid: once trained, they perform inference in a zero-shot manner and cannot adapt to the test scene. As a result, visually plausible reconstructions often contain errors, particularly under occlusions, specularities, and ambiguous cues. To address this, we introduce Free Geometry , a framework that enables feed-forward 3D reconstruction models to self-evolve at test time without any 3D ground truth . Our key insight is that, when the model receives more views, it produces more reliable and view-consistent reconstructions. Leveraging this property, given a testing sequence, we mask a subset of frames to construct a self-supervised task. Free Geometry enforces cross-view feature consistency between representations from full and partial observations, while maintaining the pairwise relations implied by the held-out frames. This self-supervision allows for fast recalibration via lightweight LoRA updates, taking less than 2 minutes per dataset on a single GPU. Our approach consistently improves state-of-the-art foundation models (e.g., Depth Anything 3 and VGGT) across 4 benchmark datasets, yielding an average improvement of 3.73% in camera pose accuracy and 2.88% in point map prediction. Code is available at: https://github.com/hiteacherIamhumble/Free-Geometry . Figure 1 : Free Geometry enables feed-forward 3D reconstruction models to self-evolve at test time without any 3D ground truth and generalize on models and datasets. 1 Introduction Recent advancements on feed-forward multi-view 3D reconstruction models, such as Depth Anything 3 [ 6 ] and VGGT [ 18 ] , have made it possible to reconstruct 3D scenes in real-time. Despite their strong zero-shot performance, these models follow a train-then-freeze paradigm: once trained on large-scale datasets, their parameters remain fixed during deployment. This zero-shot rigidity inference cannot be adjusted to test scenes. Consequently, when encountering novel test scenes, reconstructions may appear plausible yet exhibit geometric errors, especially under occlusions, specularities, and other ambiguous visual cues. A straightforward solution to better generalization is to scale training data. However, collecting large-scale, high-quality 3D ground truth across diverse real-world environments is prohibitively expensive and often impractical. This scarcity of 3D supervision also makes direct (re-)training infeasible. To improve this generalization, we instead rely on an intuitive observation: Reconstructions improve when the model sees more views. This is much expected in Table 1 , as additional viewpoints add geometric constraints and reduce ambiguity. As shown in Fig. 2 , predictions from all views are usually more accurate than those from a masked subset. This naturally suggests a self-supervision signal: the full-view prediction can serve as a teacher for the masked-view prediction. Building on this insight, we introduce Free Geometry , a test-time adaptation framework that enables feed-forward models to self-evolve when encountering test scenes. Our key idea is to use the full-view prediction of the model as a supervisor for masked-view predictions. Since full-view inputs generally produce more accurate geometry, this provides a free signal to improve the model. Moreover, we observe that the decoders in these feed-forward architectures operate on each frame independently. Therefore, Free Geometry applies supervision at the level of encoder features rather than decoder output, making the training more stable and efficient. Specifically, given a test sequence, we mask a subset of frames to form a partial-view input. The full-view input is processed by a frozen backbone to produce teacher features. The partial-view input is processed by the same backbone augmented with lightweight LoRA modules [ 5 ] to produce student features. Free Geometry then optimizes the LoRA weights by enforcing feature consistency between the teacher and student representations at the same location, while preserving the pairwise relations implied by the held-out frames. This defines a label-free, self-supervised objective to improve the geometry. Table 1: Long Sequence Provides Better Reconstruction Accuracy. Row 2 ( 8\!\rightarrow\!4 ) uses 8 input views to compute encoder features and forwards only the corresponding 4-view features to the decoder. We report pose accuracy (AUC@3 \uparrow ) and reconstruction quality (F1 \uparrow ). Rankings are highlighted with first , second , and third within each column. # Views ETH3D 7-Scenes ScanNet++ HiRoom AUC@3 \uparrow F1 \uparrow AUC@3 \uparrow F1 \uparrow AUC@3 \uparrow F1 \uparrow AUC@3 \uparrow F1 \uparrow 8 0.445 0.536 0.315 0.439 0.733 0.436 0.827 0.787 8\rightarrow 4 0.424 0.188 0.302 0.285 0.747 0.306 0.857 0.682 4 0.318 0.142 0.254 0.239 0.586 0.231 0.700 0.561 Figure 2 : Long Sequence Provides Better Reconstruction Geometry. The left example shows a HiRoom bedroom scene reconstructed by using 8 input views in VGGT while forwarding only the corresponding 4-view features to the decoder. The right example shows the same scene reconstructed using only 4 input views. As a result, Free Geometry adapts quickly at test time, with minimal overhead ( < 2 minutes per data set on a single GPU). It is a plug-and-play arbitrary feed-forward 3D reconstruction model and adds minimal computational or annotation cost. Across multiple benchmarks, it consistently improves state-of-the-art foundation models, yielding an average improvement of 3.73% in camera pose accuracy and 2.88% in point map prediction. Surprisingly, although we adapt using a single sequence setting (e.g. 8 \to 4 training), the gains transfer to different numbers of input views (e.g. 4, 8, 16 and 32). To summarize, the contributions of this work are as follows: (1) We identify a consistent “more-views-better” regime in feed-forward multi-view 3D foundation models. This provides a practical, label-free self-supervision signal to refine the model. (2) We propose Free Geometry , a plug-and-play test-time adaptation framework that performs self-supervised geometric recalibration by enforcing feature-level consistency between full-view and masked-view inputs. (3) Extensive experiments across multiple benchmarks and baselines demonstrate that our method delivers fast, low-cost adaptation and consistent improvements in both reconstruction quality and pose accuracy. 2 Related Work 2.1 Multi-View Feed-Forward 3D Reconstruction Classical multi-view stereo methods such as COLMAP [ 12 ] and MVSNet [ 20 ] rely on iterative optimization or cost volume construction, requiring known camera poses and substantial computation. Recent feed-forward approaches fundamentally change this paradigm. DUSt3R [ 19 ] introduces a transformer-based architecture that directly regresses 3D point maps from image pairs without explicit pose estimation in a purely feed-forward manner. VGGT [ 18 ] follows a similar design philosophy with a geometry-based transformer, pushing accuracy to a new level through large-scale training to achieve efficient zero-shot performance. Depth Anything 3 [ 6 ] scales to arbitrary view counts using a single ViT-Giant backbone [ 2 ] with global attention across all view tokens, jointly predicting depth, camera poses, and point maps to achieve State-of-the-Art performance. A critical architectural property shared by DA3 and VGGT is that all cross-view reasoning occurs in the backbone’s multi-view transformers, while the decoders operate entirely per-view. This property is central to Free Geometry: it identifies the backbone features as the bottleneck for test-time adaptation and motivates feature-level rather than output-level adaptation. However, these models operate under a rigid train-then-freeze paradigm, causing performance degradation with unseen test scenes, and finetuning these models with training loss is impossible without ground truth labels. 2.2 Test-Time Adaptation Test-time adaptation (TTA) adjusts a pre-trained model to the target domain using only test data, without access to the original training set. TENT [ 17 ] adapts batch normalization statistics by minimizing prediction

Read the complete text