Fast and Lightweight Novel View Synthesis with Differentiable Multiplane Image
Kaidi Zhang, Guanxu Zhu · 2026
Read Fast and Lightweight Novel View Synthesis with Differentiable Multiplane Image by Kaidi Zhang, Guanxu Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02068.txt.
Introduction
Zhang and Zhu speed up novel view synthesis with a layered image method. Novel view synthesis renders a scene from new angles. NeRF and 3D Gaussian methods give striking results. They often trade off speed against model size. Their training by optimization can also take long.
Sparse views make those methods struggle further. Feed-forward variants cut training time a lot. But they spawn millions of Gaussians per image. That limits deployment on mobile devices badly. The pair revisit the multiplane image representation. Scenes are stored as a compact set of planar layers. Differentiable rendering makes the layers trainable end to end. The design targets fast, light view synthesis. Mobile-friendly rendering is the practical goal.
Catalog record
| CREATOR | Kaidi Zhang, Guanxu Zhu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02068
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fast and Lightweight Novel View Synthesis with Differentiable Multiplane Image Kaidi Zhang, Guanxu Zhu Abstract Recently, novel view synthesis has witnessed remarkable progress, with mainstream methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) delivering impressive results. However, these approaches often struggle to balance rendering speed and model size, and their optimization-based training can be highly time-consuming. Furthermore, they typically rely on dense observations, often failing to produce satisfactory results under sparse-view conditions. Although feed-forward reconstruction significantly reduces the optimization time of 3DGS, its pixel-aligned formulation generates millions of Gaussians from a single image, severely limiting its practical deployment on mobile devices. To address these limitations, we revisit the Multiplane Image (MPI) representation, which represents scenes using a compact set of planar layers for efficient novel view synthesis. Leveraging recent advances in visual foundation models, we utilize predicted point maps for reliable geometric initialization, followed by differentiable optimization. To address the issues of holes and artifacts in sparsely initialized MPI, we introduce one-step diffusion, which participates in both the differentiable optimization of MPI and the post-processing of rendering results. Compared with a representative GS-based method, our approach is 30.7% faster and uses only 14.8% of its model size, while achieving competitive synthesis quality on front-view scenarios. 1 Introduction Sparse-view reconstruction significantly bridges the gap between 2D imagery and immersive 3D experiences, playing a crucial role in downstream applications such as robotics and 3D photography. The practical deployment of 3D representations imposes stringent requirements on both model compactness and rendering speed. Recent advances, such as Neural Radiance Fields (NeRF [ 22 ] ) and their variants, have demonstrated impressive capabilities in synthesizing high-quality novel views. However, optimizing the continuous volumetric fields of NeRF typically relies on dense multi-view observations. As an alternative explicit representation, 3D Gaussian Splatting(3DGS [ 16 ] ) has gained increasing attention due to its exceptional rendering efficiency. Despite this advantage, early 3DGS methods share a similar limitation with NeRF, requiring densely sampled viewpoints to maintain rendering fidelity. More recently, feed-forward paradigms have been introduced to directly infer 3D Gaussian representations in a single pass. While these approaches achieve notable progress, most of them adopt pixel-aligned Gaussian regression, where each pixel corresponds to one Gaussian. This design inevitably leads to severe redundancy— for instance, just a few sparse-view images can generate millions of Gaussians, rendering such methods impractical for deployment on resource-constrained devices like mobile platforms. To address the limitations arising from high resource consumption and sparse input views, we turn our attention to Multiplane Images (MPI), which provide an efficient discrete representation for modeling continuous scenes. MPI-based approaches [ 25 , 31 , 44 ] approximate a 3D scene using a set of fronto-parallel planes within a camera frustum, offering higher efficiency and lower computational cost compared to continuous representations like NeRF, albeit at the expense of representational continuity. However, existing MPI-based methods are typically restricted to the camera frustum and struggle to handle disocclusions and artifacts caused by viewpoint changes. This limitation largely stems from their reliance on feed-forward predictions (e.g., CNN-based models) from a single image, where performance is bounded by dataset scale and model capacity. Additionally, these methods are prone to depth discretization artifacts and repetitive texture patterns, which limit their applicability in real-world scenarios. Benefiting from recent advances in depth estimation and vision foundation models (e.g., VGGT [ 36 ] ), it is now possible to infer geometric structure from few images. The resulting point clouds provide a strong geometric prior for scene representation, thus enabling the differentiable optimization of MPI. Building upon these developments, we propose our Multi-view MPI, a method that leverages a learnable MPI representation. Compared to prior MPI-based approaches, our method significantly enhances 3D scene modeling capacity and supports a broader range of camera motions. Instead of hallucinating occluded regions through plane generation, which often yields low-quality texture completion, we treat MPI as a planar representation and explicitly optimize it using multi-view supervision to recover missing geometry. A key distinction of our approach lies in representing MPI as learnable parameters that are directly optimized via volume rendering, acknowledging that extended 3D scenes are too complex to be reliably inferred in a single forward pass. This formulation enables both flexible viewpoint control and efficient, 3D-consistent novel view synthesis, while effectively mitigating repetitive texture artifacts. To further address common MPI issues (e.g., artifacts), we leverage recent advances in generative models and integrate DIFIX [ 40 ] , a fast one-step diffusion model, into our pipeline in two ways: First, we interpolate sparse-view rendering results, feed them into the diffusion model for restoration, and retrain the MPI therewith to distill the strong prior into it. Second, during rendering and inference, the diffusion model acts as a post-processor to enhance rendering quality. In summary, our main contributions are as follows: • We propose Multi-view MPI, a novel view synthesis framework that enables broader-range viewpoint exploration and fast, 3D-consistent rendering. Besides, we formulate MPI as a learnable planar representation that supports the differentiable optimization from multiple views. • We incorporate a Neural Enhancer into both the optimization process and post-processing to address common MPI issues such as artifacts, thereby forming a more robust pipeline. • We conduct extensive experiments on multiple benchmarks, including the LLFF real-world dataset and the NeRF synthetic dataset. Our method achieves the best PSNR among representative novel view synthesis methods while maintaining a compact representation and high rendering efficiency. 2 Related Work 2.1 Novel View Synthesis and 3D Representation. NeRF [ 22 ] and its related works [ 9 , 4 , 1 ] have achieved remarkable progress in NVS. To achieve faster training and rendering, NeRFs have gradually incorporated explicit structures [ 3 , 23 ] , while remaining computationally expensive due to the dense sampling along each ray. In contrast, 3DGS [ 16 ] adopts a purely explicit representation, achieving photorealistic rendering quality while significantly improving rendering speed. Subsequent works have further enhanced 3DGS by incorporating techniques such as anti-aliasing [ 42 ] , training acceleration, and block-wise optimization [ 20 , 19 ] . Recently, feed-forward Gaussian methods [ 2 ] have gradually replaced the traditional reconstruction paradigm. This paradigm has achieved remarkable results in sparse-view and single-view 3D reconstruction. However, such feed-forward approaches are often pixel-aligned: each pixel is decoded into a Gaussian, which leads to severe redundancy among Gaussians. Although methods such as AnySplat [ 15 ] attempt to eliminate redundancy in multi-view pixel-aligned reconstruction using voxels, they still yield point clouds with hundreds of thousands of Gaussians. Consequently, the aforementioned 3D representations impose high requirements on computing devices. 2.2 Layer-based Methods Layer-based methods represent a 3D scene using discrete layers, resulting in higher efficiency and lower computational cost compared to continuous 3D representations like NeRF. Layer-based methods, such as 3D-photography [ 26 ] and SLIDE [ 14 ] , can produce high-quality synthesis results from a single input image. As a discrete diffuse volumetric scene representation, MPI-based methods [ 25 , 31 , 44 ] were proposed to represent the scene using multiple planes in a frustum. Srinivasan et al. [ 30 ] proposed a two-step MPI prediction procedure to alleviate issues of depth discretization artifacts and repeated texture artifacts that