On-the-fly Reconstruction for Large-Scale Novel View Synthesis from Unposed Images 1024 Journal: TOG Volume: 44 4 8 DOI: 10.1145/3730913 CCS: Computing methodologies Rasterization CCS: Computing methodologies Point-based models CCS: Computing methodologies Matching CCS: Computing methodologies Recon
Andreas Meuleman, Ishaan Shah, Alexandre Lanvin, Bernhard Kerbl, George Drettakis · 2025
Read On-the-fly Reconstruction for Large-Scale Novel View Synthesis from Unposed Images 1024 Journal: TOG Volume: 44 4 8 DOI: 10.1145/3730913 CCS: Computing methodologies Rasterization CCS: Computing methodologies Point-based models CCS: Computing methodologies Matching CCS: Computing methodologies Recon by Andreas Meuleman, Ishaan Shah, Alexandre Lanvin, Bernhard Kerbl, George Drettakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05558.txt.
Introduction
Andreas Meuleman and four co-authors rebuild three-dimensional scenes from photos on the fly, with no wait after shooting. Their method appeared in twenty twenty-five. Viewpoint-free rendering from photos is now easy, but finding camera poses and training the scene model can take minutes to hours.
Their approach estimates poses fast, using learned image features and a small GPU-friendly adjustment step. It then seeds scene blobs directly where needed and grows them during training, which speeds the fit sharply. For large scenes, blobs are grouped into anchors and parked off the graphics card, then merged as work proceeds. Tests across many data sets show the method matches rivals in quality while delivering a walkable scene as soon as capture ends.
Catalog record
| CREATOR | Andreas Meuleman, Ishaan Shah, Alexandre Lanvin, Bernhard Kerbl, George Drettakis |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05558
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On-the-fly Reconstruction for Large-Scale Novel View Synthesis from Unposed Images 1024 Journal: TOG Volume: 44 4 8 DOI: 10.1145/3730913 CCS: Computing methodologies Rasterization CCS: Computing methodologies Point-based models CCS: Computing methodologies Matching CCS: Computing methodologies Reconstruction CCS: Computing methodologies Tracking Andreas Meuleman, Ishaan Shah, Alexandre Lanvin, Bernhard Kerbl, George Drettakis Abstract . Radiance field methods such as 3D Gaussian Splatting (3DGS) allow easy reconstruction from photos, enabling free-viewpoint navigation. Nonetheless, pose estimation using Structure from Motion and 3DGS optimization can still each take between minutes and hours of computation after capture is complete. SLAM methods combined with 3DGS are fast but struggle with wide camera baselines and large scenes. We present an on-the-fly method to produce camera poses and a trained 3DGS immediately after capture. Our method can handle dense and wide-baseline captures of ordered photo sequences and large-scale scenes. To do this, we first introduce fast initial pose estimation, exploiting learned features and a GPU-friendly mini bundle adjustment. We then introduce direct sampling of Gaussian primitive positions and shapes, incrementally spawning primitives where required, significantly accelerating training. These two efficient steps allow fast and robust joint optimization of poses and Gaussian primitives. Our incremental approach handles large-scale scenes by introducing scalable radiance field construction, progressively clustering 3DGS primitives, storing them in anchors, and offloading them from the GPU. Clustered primitives are progressively merged, keeping the required scale of 3DGS at any viewpoint. We evaluate our solution on a variety of datasets and show that it can provide on-the-fly processing of all the capture scenarios and scene sizes we target. At the same time our method remains competitive – in speed, image quality, or both – with other methods that only handle specific capture styles or scene sizes. (a) Initialization (b) On-the-fly 3DGS (c) H3DGS: 22hrs, PSNR 12 dB (d) Ours: 25min, PSNR 21.7 dB Figure 1. Our method performs on-the-fly reconstruction from an unposed, ordered image sequence. The total processing time for our method is 30min for this sequence of over 4000 images: with our method, the poses and radiance field are immediately available after taking the photos. The scene was captured in a 1km walk in 30min, using a camera in "drive mode" taking photos at 3 images/sec, resulting in around 4000 images. Our method incrementally reconstructs a 3D Gaussian representation along with the camera poses. In the middle right, we show a novel view of the same scene reconstructed with Hierarchical 3DGS ( Kerbl et al. 2024 ) , that requires 22 hours of processing for camera pose estimation and 3DGS optimization, in contrast to our method (right) that has completed all processing by the time photos are taken. The camera calibration of hierarchical 3DGS fails in many places later on the path, leading to novel view synthesis failure in some places (please see video and additional results at https://repo-sam.inria.fr/nerphys/on-the-fly-nvs/ ). † † cc-license: by 1. Introduction Radiance Field solutions for novel-view synthesis ( Barron et al. 2022 ; Müller et al. 2022 ; Duckworth et al. 2023 ) take a multi-view dataset of images as input and create a 3D digital version of a real scene, enabling free-viewpoint navigation. Since the recent introduction of 3D Gaussian Splatting (3DGS) ( Kerbl et al. 2023 ) – a fast radiance field method with high visual quality – there has been an explosion in the application of radiance fields in domains as diverse as e-commerce, extended reality, film and video, robotics, 3D generative modelling etc. Most such solutions first need to perform pose estimation for all cameras upfront ( Schönberger and Frahm 2016 ) followed by 3DGS optimization; for typical wide-baseline datasets (tens of) minutes are still required for each of the two steps, even with the fastest methods ( Mallick et al. 2024 ) . For larger scenes, these steps each require many hours of processing after capture has finished, hindering the adoption of such solutions in applications. We present a method for on-the-fly radiance field construction that works on consumer hardware: with our solution, camera poses and the scene representation are immediately available by the time scene capture is finished. While Structure-from-Motion (SfM) ( Schönberger and Frahm 2016 ) followed by 3DGS ( Kerbl et al. 2023 ; Mallick et al. 2024 ) are too slow for on-the-fly processing, SLAM solutions are much faster ( Tosi et al. 2024 ) , but typically expect dense, video-like input, and often focus less on visual quality. These solutions often struggle with wide-baseline multi-view captures, typically used for radiance-field reconstruction. Similarly, 3DGS optimization is still too costly for on-the-fly reconstruction. 3DGS optimization starts from a sparse, noisy set of SfM points, while the densification process provides additional points from heuristics. As a result, the optimization needs to proceed carefully, requiring many iterations. This results in long training times, even for the most efficient versions ( Mallick et al. 2024 ) . Finally, while some solutions have been proposed to allow 3DGS for large scenes ( Ren et al. 2024 ; Kerbl et al. 2024 ; Liu et al. 2024a ) , they all require the poses of all cameras to be estimated beforehand, for scene subdivision and/or hierarchy construction. Our goals are to provide both camera calibration and usable 3DGS reconstruction immediately after capture, either for dense or wide-baseline capture, and most importantly for large-scale scenes. Other methods have difficulty satisfying all of these goals. Note that we require captured images to be given in an ordered sequence, since we present an incremental radiance field construction method. We observe that accurate pose estimation can be costly; if we relax the need for accuracy in a first step, we can reformulate pose estimation to be GPU-friendly and thus significantly faster even if the initial guess is approximate. For such an approach to be possible, joint pose/3DGS estimation needs to subsequently refine these poses. 3DGS differentiable rendering is well-suited to this task, since the gradients from the rendering loss can flow to the poses, improving their accuracy. Finally, for joint optimization to be effective, we also need to accelerate 3DGS; a good strategy to achieve this goal is to directly sample well placed Gaussians, reducing densification and making the optimization easier. Given these observations, we first propose lightweight initial pose estimation that leverages learning-based matches with image neighbors. The matches and their currently optimized poses are used to quickly estimate the next camera pose. We also reformulate the problem to allow fast GPU-friendly mini bundle adjustment. Second, instead of densifying, we introduce a method for direct sampling of Gaussian primitives. When adding a new image, we directly choose the positions and sizes of 3D Gaussian primitives created, by estimating the probability that a given pixel should spawn a Gaussian, and also directly sample the size of each primitive. This greatly reduces the need for densification, providing the required optimization speed. Given our incremental pose estimation and training, joint optimization is accelerated and the risk of getting stuck in local minima is reduced. Finally, our efficient incremental optimization is naturally suited to a sliding-window clustering and merging approach which we propose, that stores parts of the scene as anchors that are placed in space as image capture advances. Taken together, these elements allow