FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation

Wenbin Teng, Gonglin Chen, Haiwei Chen, Yajie Zhao · 2025

Read FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation by Wenbin Teng, Gonglin Chen, Haiwei Chen, Yajie Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06392.txt.

Introduction

Wenbin Teng and co-authors make three-dimensional view generation far faster with adversarial distillation, in work from 2025. Rebuilding a scene from few photos leaves unseen areas full of artifacts. Video diffusion models can fill those gaps, but they sample painfully slowly.

Their framework, FVGen, compresses a many-step teacher model into a student needing four steps. Training mixes adversarial loss with a softened divergence target. On real data sets, the student matched prior quality while cutting sampling time by over ninety percent. The same count of new views came out at a fraction of the cost. Faster fills make sparse capture practical for games and scanning alike.

Catalog record

CREATORWenbin Teng, Gonglin Chen, Haiwei Chen, Yajie Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06392

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation Wenbin Teng, Gonglin Chen, Haiwei Chen, Yajie Zhao Abstract Recent progress in 3D reconstruction has enabled realistic 3D models from dense image captures, yet challenges persist with sparse views, often leading to artifacts in unseen areas. Recent works leverage Video Diffusion Models (VDMs) to generate dense observations, filling the gaps when only sparse views are available for 3D reconstruction tasks. A significant limitation of these methods is their slow sampling speed when using VDMs. In this paper, we present FVGen, a novel framework that addresses this challenge by enabling fast novel view synthesis using VDMs in as few as four sampling steps. We propose a novel video diffusion model distillation method that distills a multi-step denoising teacher model into a few-step denoising student model using Generative Adversarial Networks (GANs) and softened reverse KL-divergence minimization. Extensive experiments on real-world datasets show that, compared to previous works, our framework generates the same number of novel views with similar (or even better) visual quality while reducing sampling time by more than 90%. FVGen significantly improves time efficiency for downstream reconstruction tasks, particularly when working with sparse input views (more than 2) where pre-trained VDMs need to be run multiple times to achieve better spatial coverage. Project Page: https://wbteng9526.github.io/fvgen/ 1 Introduction Reconstructing 3D scenes from 2D images remains a pivotal research area at the intersection of computer vision and graphics. This field has extensive social impact, with broad applications in autonomous navigation, augmented reality, and more. With the recent development of neural rendering techniques such as NeRF [ 29 ] and 3DGS [ 16 ] , creating high-quality 3D assets from 2D images has become more accessible. However, the success of 3D reconstruction depends on dense observations, which are often difficult to obtain in real-world scenarios. This limitation makes it challenging to apply these methods in many practical applications. Introducing generative models into 3D reconstruction can fill gaps in missing views when only sparse input views are available. However, novel view synthesis from sparse views, especially in the settings with less than 2 views, is inherently an ill-posed problem. A bulk of previous works [ 52 , 43 , 30 , 7 , 8 ] have found that priors learned from large image and video diffusion models [ 10 , 11 ] are rich enough to inpaint detailed and reasonable information in the many under-observed areas in the reconstructed 3D representations. We are primarily interested in the use of video diffusion models (VDM) to perform novel view synthesis. What bridges VDM to novel view synthesis is its ability to synthesize continuous views following a camera trajectories between the observed views. The spatio-temporal consistency in these smooth video trajectories, as demonstrated in several works [ 52 , 7 , 44 , 6 , 25 ] , effectively addresses the challenging ”Janus Problem” that typically limits generation quality in other multi-view settings. However, VDM is a computationally expensive method because of the iterative sampling process that characterized all denoising diffusion methods. The long generation time makes VDM-based generation methods unsuitable for certain real-world downstream tasks, such as dynamic 3D reconstruction, where new consistent views must be generated continuously. This limitation also affects large-scale 3D reconstruction that requires synthesizing a significant number of novel views. In this paper, we show that VDM-based novel view synthesis can be significantly accelerated for free. Without any compromise on the generation quality, we speed up diffusion-based synthesis of novel view by a factor of 90% with a 4-step student model trained under a GAN objective [ 23 ] . Acceleration of diffusion models has been actively studied in the past, but previous techniques may not be well suited to our task due to two reasons: First, we need a method specifically for accelerating video diffusion models, rather than image diffusion models. Second, the method must generalize effectively when trained on multi-view video datasets, which contain significantly less data than general video datasets. To accelerate a diffusion model, recent methods [ 49 , 34 , 28 , 22 , 46 ] propose minimizing the distributional difference between a few-step student generation model and the original multi-step teacher model. However, these methods are restricted to images with specific domains. The most related work to us is [ 50 ] , which generates more video frames with faster speed by optimizing a distribution matching distillation (DMD) loss. However, this method requires an effective initialization of the student model. This is completed by generating a noise-latent pair through an ODE solver [ 36 ] and training the student model by optimizing a regression loss [ 49 , 50 ] . The generation process usually takes a very long time and the student’s ability is limited by the teacher’s generation quality. More importantly, we have observed that DMD optimizations tend to be unstable and have the tendency towards mode collapse, when trained under multi-view video data. This may be due to the inherent concept of DMD loss optimization being the reverse KL-divergence minimization, which tends to be mode-seeking and zeroing out the modes of teacher’s that are not relatively dominant. Figure 1 : Overview of student initialization. We initialize our student model by training a GAN objective. The student model G_{\theta} , initialized with the weights of teacher model, uses few-step denoising to generate fake videos to fool the discriminator D . The fixed teacher model, together with a trainable discriminator, are optimized to differentiate between fake and real videos. To solve the previous problems with VDM acceleration, we propose a novel framework, named FVGen , that performs novel views generation with a fast video diffusion generation with as few as 4 steps. First of all, different from previous GAN-based distillation methods [ 51 , 34 ] , we find it beneficial to initial the few-step student model by training a GAN objective, where the student model is regarded as a generator of fake samples, and, instead of training a separate discriminator as in previous works, we leverage a pre-trained teacher model as the discriminator. Second, we propose the soften reverse KL-divergence to solve the unstable optimization of DMD loss. The soften reverse KL-divergence maintains similar mode-seeking behavior as reverse KL-divergence but with more robust distillation: it prevents the student from ignoring entire space of teacher’s distribution, thereby preserving more of teacher’s knowledge, which we find particularly beneficial in the context of limited data. Through effective and robust distillation, FVGen is capable of generating video sequences with comparable or better visual quality as ViewCrafter [ 52 ] but with significantly less time. FVGen is trained with Dl3DV-10K [ 24 ] dataset and extensive experiments on real world dataset including MipNeRF360 [ 1 ] and Tanks-and-Temples [ 18 ] demonstrate that our method achieves overall better performance than other state-of-the-art (SOTA) novel view synthesis methods and diffusion distillation methods. 2 Related Works 2.1 Diffusion-based Novel View Synthesis NeRF [ 29 ] and 3DGS [ 16 ] excel at novel view synthesis with dense input views. However, reconstructing scenes from sparse views requires additional priors. Diffusion models [ 10 , 32 ] can generate realistic pseudo input views from sparse original inputs. Several works use Score-Distillation Sampling [ 30

Read the complete text