AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model

Yutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang, Xin Cai, Yawen Luo, Mingxin Yang, Mulin Yu, Linning Xu, Tianfan Xue · 2026

Read AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model by Yutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang, Xin Cai, Yawen Luo, Mingxin Yang, Mulin Yu, Linning Xu, Tianfan Xue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19747.txt.

Introduction

Chen and nine colleagues present AnyRecon, a scalable tool for 3D rebuilding from arbitrary, unordered sparse views that keeps direct geometric control. Earlier diffusion methods synthesize new views but use only one or two capture frames, which limits shape consistency and scale to large or diverse scenes. AnyRecon builds a persistent global scene memory from a prepended capture view cache and drops time-based compression, preserving frame-level correspondence.

Catalog record

CREATORYutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang, Xin Cai, Yawen Luo, Mingxin Yang, Mulin Yu, Linning Xu, Tianfan Xue
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39613
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19747

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AnyRecon: Arbitrary-View 3D Reconstruction with Video Diffusion Model Yutian Chen, Shi Guo, Renbiao Jin, Tianshuo Yang, Xin Cai, Yawen Luo, Mingxin Yang, Mulin Yu, Linning Xu, Tianfan Xue Abstract Sparse-view 3D reconstruction is essential for modeling scenes from casual captures, but remain challenging for non-generative reconstruction. Existing diffusion-based approaches mitigates this issues by synthesizing novel views, but they often condition on only one or two capture frames, which restricts geometric consistency and limits scalability to large or diverse scenes. We propose AnyRecon, a scalable framework for reconstruction from arbitrary and unordered sparse inputs that preserves explicit geometric control while supporting flexible conditioning cardinality. To support long-range conditioning, our method constructs a persistent global scene memory via a prepended capture view cache, and removes temporal compression to maintain frame-level correspondence under large viewpoint changes. Beyond better generative model, we also find that the interplay between generation and reconstruction is crucial for large-scale 3D scenes. Thus, we introduce a geometry-aware conditioning strategy that couples generation and reconstruction through an explicit 3D geometric memory and geometry-driven capture-view retrieval. To ensure efficiency, we combine 4-step diffusion distillation with context-window sparse attention to reduce quadratic complexity. Extensive experiments demonstrate robust and scalable reconstruction across irregular inputs, large viewpoint gaps, and long trajectories. 0 0 footnotetext: † indicates corresponding author. 1 Shanghai AI Lab 2 CUHK MMLab 3 Shanghai Jiao Tong University 4 The University of Hong Kong 5 CPII under InnoHK https://yutian10.github.io/AnyRecon/ 1 Introduction Figure 1 : AnyRecon demonstrates robust performance across multiple reconstruction settings: (Top) Interpolation, filling in gaps between distant captured views; (Middle) Extrapolation, synthesizing novel content beyond the observed range; and (Bottom) Large Scene Reconstruction, maintaining consistency across long-trajectory sequences. The camera visualization (left) illustrates the sparse input poses (red) and the dense generated path (blue) . Novel view synthesis and 3D reconstruction are fundamental problems in computer vision and graphics, enabling applications ranging from immersive virtual environments to augmented reality and visual effects. Recent advances in neural reconstruction methods, including implicit representations such as NeRF [ 14 ] and explicit point-based approaches like 3D Gaussian Splatting [ 8 ] , have demonstrated remarkable visual fidelity. However, these methods rely on densely sampled multi-view images captured under controlled acquisition setups. While real-world visual data—such as handheld captures or Internet videos—are typically sparse and irregular. Enabling reconstruction from such arbitrary view would allow scalable conversion of everyday captures into explorable 3D scenes. Recent work try to mitigate this sparse view challenge by using more views created by diffusion-based novel view synthesis. While early efforts employ image generative models to infer 3D structure [ 24 , 23 ] , more recent works leverage video generation models to synthesize novel views, as they can better capture cross-view coherence through temporal modeling. One line of work [ 29 , 16 , 2 ] conditions diffusion models primarily on projected point cloud renderings, providing only one or two captured RGB frames (e.g., the first and last views). Although renderings offer coarse geometric guidance, limited real-image conditioning weakens appearance fidelity and global context, making generation sensitive to incomplete or low-quality geometry. Another line of work [ 1 , 6 ] relies solely on RGB images and camera poses without explicitly incorporating reconstructed geometry into the generation process. By learning geometric consistency implicitly, these methods struggle to maintain precise pose alignment and spatial consistency, limiting their deployment for real-world 3D reconstruction under irregular observations. In this work, we aim to enable high-quality and large scale 3D reconstruction from sparse inputs. Unlike prior sparse-view diffusion models restricted to only one or two reference views, our diffusion model flexibly conditions on an arbitrary number of captured RGB images alongside point cloud renderings. However, supporting flexible input cardinality introduces many challenges. First, input images may be arbitrary captured with large viewpoint gaps, while existing video diffusion frameworks are suboptimal for non-sequential inputs, as they rely on temporally causal latent compression [ 12 ] . Moreover, to reconstruct a large complex scene, it is impossible to fit all input into diffusion model at once. Therefore, a robust iterative reconstruction strategy is required to reconstruct a scene by small segment. These challenges demand a reconstruction framework that preserves fine-grained control while remaining computationally efficient. To address these challenges, we propose AnyRecon, a scalable framework for sparse-view 3D reconstruction. First, we develop a diffusion-based novel view synthesis model that supports arbitrary and unordered sparse inputs while maintaining explicit geometric control. Specifically, we construct a 3D point cloud from the sparse captures and render it into target viewpoints to serve as geometric conditions. To handle flexible inputs, we maintain a global memory cache by prepending these original captures to the rendered priors in the sequence, thereby enabling long-range conditioning across arbitrary viewpoints. Furthermore, to ensure frame-level correspondence under large viewpoint changes, we remove temporal compression in the latent encoder. At last, to reduce computation for large-scale reconstruction, we adopt a 4-step diffusion distillation strategy and introduce a context-window sparse attention mechanism that restricts attention to local temporal windows and geometry-aligned retrieved views. Furthermore, to support segment-by-segment reconstruction of a large-scale scene, we introduce a geometry-aware conditioning strategy that couples generation and reconstruction. Specifically, this strategy creates an iterative loop where generated outputs continuously update a shared 3D geometry, which in turn guides the conditioning of subsequent segments. First, we build an explicit 3D Geometry Memory by back-projecting newly generated images into the initial point cloud, enabling incrementally updated geometric memory across trajectory segments. Second, when conditioning a new trajectory segment, we perform geometry-driven view selection from a captured view bank based on geometric contribution and spatial overlap with the current reconstruction, rather than relying on image-level similarity or field-of-view (FOV) heuristics [ 28 , 10 , 3 ] . Together, these components form a closed geometric loop between reconstruction and generation, ensuring that diffusion is guided by spatially informative observations and improving robustness under occlusion and complex scene layouts. Extensive benchmarks demonstrate that AnyRecon delivers superior results compared to state-of-the-art baselines. As shown in Fig. 1 , our method facilitates seamless view interpolation and long-range extrapolation across diverse, large-scale scenes (over 200 frames), maintaining high fidelity despite sparse and irregular input captures. Our key contributions are summarized as follows: • A flexible sparse-view reconstruction framework . We propose a video-diffusion-based approach that supports arbitrary and unordered conditioning views, enabling robust 3D reconstruction. • A geometry-aware conditioning design. We couple generation and reconstruction via a 3D Geometry Memory with back-projection and geometry-driven capture-view retrieval, ensuring spatially grounded diffusion guidance. • An efficient diffusion architecture. By removing temporal compression and adopting diffusion distillation with block sparse attention, our method generalizes across varying numbers of input views while maintaining computational efficiency. 2 Related Work 2.1 Traditional Sparse-View Reconstruction. Sparse-view reconstruction is inherently ill-posed due to the vast unobserved regions and geometric ambiguities arising from limited inputs. Early efforts addressed this by incorporating various geometric priors and regularization techniques. For instance, FreeNeRF [ 25 ] and RegNeRF [ 15 ] employ frequency-domain regularization and depth smoothness constraints to stabilize optimization when only sparse views are available. Other approaches focus on leveraging auxiliary supervision from pre-trained models to provide additional scene constraints. Specifically, SPARF [ 17 ] utilizes correspondence field and

Read the complete text