ERUPT: Efficient Rendering with Unposed Patch Transformer
Maxim V. Shugaev Vincent Chen Maxim Karrenbach Kyle Ashley Bridget Kennedy Naresh P. Cuntoor * BlueHalo · 2025
Read ERUPT: Efficient Rendering with Unposed Patch Transformer by Maxim V. Shugaev Vincent Chen Maxim Karrenbach Kyle Ashley Bridget Kennedy Naresh P. Cuntoor * BlueHalo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-24374.txt.
Introduction
Maxim Shugaev and five co-authors present ERUPT, a transformer model that renders new views of a scene from as few as five unposed photographs, in a two thousand twenty-five computer vision paper. Methods like NeRF and Gaussian splatting demand dense imagery and precise camera metadata.
ERUPT replaces pixel-based queries with patch-based querying, cutting compute sharply during training and inference. It renders at six hundred frames per second on commercial hardware. A learned latent camera pose lets it train on datasets with sparse or inaccurate ground truth poses. The team also introduces MSVS-1M, a street-view benchmark from Mapillary. ERUPT beats current methods on unposed synthesis while needing far fewer labeled images.
Catalog record
| CREATOR | Maxim V. Shugaev Vincent Chen Maxim Karrenbach Kyle Ashley Bridget Kennedy Naresh P. Cuntoor * BlueHalo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-24374
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ERUPT: Efficient Rendering with Unposed Patch Transformer Maxim V. Shugaev Vincent Chen Maxim Karrenbach Kyle Ashley Bridget Kennedy Naresh P. Cuntoor * BlueHalo Abstract This work addresses the problem of novel view synthesis in diverse scenes from small collections of RGB images. We propose ERUPT (Efficient Rendering with Unposed Patch Transformer) a state-of-the-art scene reconstruction model capable of efficient scene rendering using unposed imagery. We introduce patch-based querying, in contrast to existing pixel-based queries, to reduce the compute required to render a target view. This makes our model highly efficient both during training and at inference, capable of rendering at 600 fps on commercial hardware. Notably, our model is designed to use a learned latent camera pose which allows for training using unposed targets in datasets with sparse or inaccurate ground truth camera pose. We show that our approach can generalize on large real-world data and introduce a new benchmark dataset (MSVS-1M) for latent view synthesis using street-view imagery collected from Mapillary. In contrast to NeRF and Gaussian Splatting, which require dense imagery and precise metadata, ERUPT can render novel views of arbitrary scenes with as few as five unposed input images. ERUPT achieves better rendered image quality than current state-of-the-art methods for unposed image synthesis tasks, reduces labeled data requirements by 95% and decreases computational requirements by an order of magnitude, providing efficient novel view synthesis for diverse real-world scenes. 1 Introduction Recovering 3D scene representation from a few images is a fundamental challenge in computer vision with wide-reaching applications in navigation, robotics, and augmented reality. Recent advances in learning implicit ( e.g. NeRF [ 21 ] ) and explicit ( e.g. Gaussian Splatting [ 13 ] ) scene representation have shown impressive capability in rendering high-fidelity, novel views of complex environments. These models typically rely on dense imagery with precise camera pose to reconstruct highly accurate models which can render novel views of the scene. However, one problem, which is central to both the NeRF and Gaussian Splatting frameworks, is that a new network or a set of Gaussians must be trained for each scene. This requires significant training time and limits the model’s ability to incorporate prior knowledge in cases such as limited input views or large camera shifts. Several methods have been introduced to reduce this scene dependency, by including depth [ 8 ] , normal [ 32 ] , semantic [ 5 ] and other priors to use information from external data as a basis for learning a particular scene. Yet the problem remains that these frameworks are generally unable to perform novel view synthesis without training on a sufficiently large number of input views and associated camera information for each new scene. More recent work has explored techniques for generalized 3D reconstruction with limited imagery using a latent scene representations transformer (SRT) [ 28 ] and learnable pose estimator (RUST) [ 27 ] to reduce reliance on precise camera poses. By training on large and diverse datasets these models learn to reconstruct arbitrary scenes using fewer images and limited camera pose information. While SRT allows images to be queried directly using an arbitrary camera pose, it does not provide a mechanism for learning from unposed target views. This makes it difficult to apply in datasets where ground truth camera pose is noisy or unknown. RUST addresses this limitation using a learnable pose estimator but requires a portion of the target image to query the model, which limits control of the camera at inference. Both methods require significant compute during training and inference using individual queries to the decoder for each pixel in the rendered images. Furthermore, existing benchmark datasets of sufficient scale are either composed of synthetic data (MSN) or are proprietary datasets not available to the public, making verification and real-world application of these techniques difficult. To address these limitations we propose ERUPT, a computationally effective approach for novel view synthesis trained on posed and unposed target images, providing direct camera control at inference. Our model includes a learnable latent camera pose which is sampled in conjunction with sparse ground truth pose to address cases of sparse or inaccurate target camera pose. We introduce a novel decoding strategy using patch-based rays instead of pixel-based rays to improve computation efficiency of rendering by an order of magnitude over existing models. To accelerate convergence, during training we generate multiple targets reusing scene representation and apply a pretrained feature extractor for transfer learning. We compare our model with existing methods using the MSN benchmark dataset, achieving state-of-the-art performance even compared to methods which rely on 100% posed target images. Additionally, we introduce a new real-world view synthesis dataset called MSVS-1M (Mapillary Street-view Synthesis 1-million) to allow future studies to conduct direct comparisons on real-world data. In summary our contributions include: 1. An efficient network architecture and training regime for latent-based novel view synthesis trained on posed and unposed target images which provides direct camera control at inference. We show that training on data with only 5% posed targets results in minimal performance degradation compared to the baseline with target poses provided. 2. Introduction of patch-based rays in the decoder instead of pixel-based rays results in an order of magnitude reduction in rendering and training time as well as VRAM requirements. Specifically, the synthesis of 224×224 images can be performed at a rate of over 600 fps. 3. An optimized model architecture for image synthesis with unknown input pose. The produced models establish a new SOTA for image synthesis on MSN dataset with unknown input pose in terms of PSNR and SSIM. 4. Using ERUPT as a foundation, we investigate the use of adversarial learning and diffusion-based rendering for latent novel view synthesis. Both approaches drastically improve the perceptual quality of the output and outperform previous perception based SOTA on MSN dataset. 5. A new multi-scene view synthesis dataset, MSVS-1M, curated from Mapillary imagery to facilitate large-scale experiments in generalized real-world view synthesis. Figure 1 : Schematic illustration of ERUPT model. B refers to backbone (feature extractor), while SA and CA refer to self- and cross- attention. 2 Related Work 2.1 NeRF and Gaussian Splatting In the realm of novel view synthesis, two main frameworks have stood out at the forefront. Neural Radiance Fields (NeRF) [ 21 ] has approached the problem by predicting the color and density of points in a 3D representation space through neural networks as an implicit representation. By modifying the neural network and training scheme, this framework can serve as a base for further works, like object editing/removal [ 14 ] , appearance or lighting editing [ 20 ] , or dynamic scene synthesis [ 2 ] . More recently, 3D Gaussian Splatting [ 13 ] has been proposed and shown to reduce the high rendering time cost of NeRFs at the cost of requiring most detailed hyperparameters. The 3DGS framework uses an explicit representation of the 3D scene based on 3D Gaussians with spherical harmonics defining the color and opacity. The low rendering time is a draw to this type of framework for inference-heavy applications, and the granularity of the 3D Gaussians allow for techniques such as per-Gaussian editing [ 3 ] , Gaussian grouping [ 40 ] , or appearance