SIR-DIFF: Sparse Image Sets Restoration with Multi-View Diffusion Model

Yucheng Mao *, Boyang Wang *, Nilesh Kulkarni, Jeong Joon Park · 2025

Read SIR-DIFF: Sparse Image Sets Restoration with Multi-View Diffusion Model by Yucheng Mao *, Boyang Wang *, Nilesh Kulkarni, Jeong Joon Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14463.txt.

Introduction

Yucheng Mao and co-authors restore photos by reasoning over several degraded shots of one scene at once in a 2025 paper. Single-image restoration guesses at missing detail, an ill-posed puzzle. Multiple photos of the same place each lose different information, so together they constrain the answer far better.

Their multi-view diffusion model denoises all views jointly, mining the relationships between them for shared structure. It beat leading single-image methods and even video-based ones on deblurring and super-resolution tests. Because the outputs agree in 3D, the cleaned sets plug straight into reconstruction and pose estimation pipelines. The authors frame sparse photo sets, like a tourist's handful of snaps, as an underexploited restoration source.

Catalog record

CREATORYucheng Mao *, Boyang Wang *, Nilesh Kulkarni, Jeong Joon Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SIR-DIFF: Sparse Image Sets Restoration with Multi-View Diffusion Model Yucheng Mao *, Boyang Wang *, Nilesh Kulkarni, Jeong Joon Park Abstract Figure 1 : Sparse View Image Restoration . Our diffusion model takes multi-view images and jointly enhances their visual quality while maintaining 3D consistency. (Top) Four motion-blurred input images are processed by our method, resulting in sharp outputs that significantly outperform single-view restoration methods as shown in the corresponding purple boxes. (Bottom) Our method can consistently restore multi-view images (4 out of 50 shown), leading to accurate 3D reconstructions. † † footnotetext: * Equal contribution Abstract The computer vision community has developed numerous techniques for digitally restoring true scene information from single-view degraded photographs, an important yet extremely ill-posed task. In this work, we tackle image restoration from a different perspective by jointly denoising multiple photographs of the same scene. Our core hypothesis is that degraded images capturing a shared scene contain complementary information that, when combined, better constrains the restoration problem. To this end, we implement a powerful multi-view diffusion model that jointly generates uncorrupted views by extracting rich information from multi-view relationships. Our experiments show that our multi-view approach outperforms existing single-view image and even video-based methods on image deblurring and super-resolution tasks. Critically, our model is trained to output 3D consistent images, making it a promising tool for applications requiring robust multi-view integration, such as 3D reconstruction or pose estimation. Project website: https://myc634.github.io/sirdiff/ 1 Introduction Image restoration approaches [ 56 , 8 , 16 , 31 ] typically process degraded images in isolation, confronting the challenge of the original scene content. Yet in practice, we frequently capture multiple photographs of meaningful moments, yielding a natural collection of views of the same scene, albeit some degraded. This common behavior presents an untapped opportunity: leveraging multiple captures to enhance image quality and better preserve our visual memories. In this work, we present a novel perspective on image restoration by reformulating it as a multi-view collaborative restoration task. We leverage our key insight that multiple corrupted photographs of the same scene contain complementary information that can aid superior denoising outcomes. The concept of multi-image fusion for quality enhancement has been explored in classical super-resolution techniques where sub-pixel shifted images are combined to reconstruct fractional pixel information [ 20 ] . Our approach revisits this paradigm in a fundamentally wider context by using neural attention mechanisms to implicitly fuse overlapping information from general multi-view images to enhance even severe aberrations. Notably, our multi-view restoration framework has particular significance for 3D computer vision applications. Many real-world tasks, from robotic SLAM to novel-view synthesis, rely fundamentally on multi-view imagery of a scene. When restoring degraded images within such multi-view sets, maintaining geometric consistency across views is crucial to preserve the underlying 3D scene assumptions. Traditional single-view restoration methods, inevitably introduce inconsistencies that can compromise these geometric constraints. In contrast, our restoration approach naturally enforces view consistency through learned neural priors. We implement our approach, dubbed SIR-Diff ( S parse I mage R estoration with Diff usion), through an innovative multi-view diffusion model architecture. Our choice of diffusion models is motivated by their natural handling of uncertainty in image restoration. We extend the UNet-based Stable Diffusion [ 37 ] architecture to process multiple views jointly by introducing three key components: (1) a unified encoder for diverse degraded inputs including low-resolution, blurred, and regular RGB images, (2) a hybrid 2D-3D convolutional structure derived from SVD (Stable Video Diffusion) [ 5 , 4 ] , and (3) a 3D self-attention Transformer that enables cross-view attention across all spatial and multi-view tokens. We evaluate and show zero-shot performance on Scannet++ [ 64 ] , ETH3D [ 40 ] , and CO3D [ 34 ] for multi-view deblurring and super-resolution tasks and their 3D downstream applications. SIR-Diff consistently outperforms existing single-image and video restoration methods for multi-view restoration tasks in both image quality and our proposed 3D view consistency metrics. For downstream applications, we show that our multi-view restoration strategy significantly enhances 3D reconstruction quality, as evidenced by improved novel-view synthesis accuracy and feature-matching success rates. Beyond image restoration, our architecture shows promise in other multi-view tasks, notably achieving competitive depth estimation accuracy and geometric consistency compared to leading single-view methods. 2 Related Works Image Diffusion Models. Diffusion Models (DM) show huge popularity in all domains: illustrations [ 68 , 54 , 9 , 37 ] , video creation [ 4 , 32 , 5 ] , 3D modeling with limited input views [ 29 , 43 , 48 ] , 4D generation [ 27 , 2 , 3 ] , and etc . Most models are fine-tuned from the pre-trained weight instead of training from the beginning. One of the most widely-used pre-trained Diffusion Models is Stable Diffusion 2.1 [ 37 ] , a UNet-based [ 38 ] architecture. Multiview-Image Diffusion models. In the image-to-3D task [ 25 ] , Diffusion Model is a powerful generative prior to enriching details for the novel (unseen) view synthesis [ 17 , 6 , 59 , 58 , 47 ] . Methods like MVDream [ 43 ] , CAT3D [ 13 ] , Zero123++ [ 42 ] jointly generate multi-views simultaneously by extending the image diffusion model (SD2.1) with a 3D self-attention layer. However, these works ignore the exploration of the convolution layer architecture in the UNet-based Diffusion Model. In this paper, we explore the methodology of applying the 3D self-attention layer to the 3D restoration task. Image and Video Restoration. These methods aim at converting a degraded input image or video into a high-quality form. Traditional tasks include Super-Resolution [ 24 , 56 , 65 , 60 , 71 ] , Deblurring [ 26 , 23 , 33 ] , Denoise [ 62 , 10 ] , Compression Artifact Reduction [ 8 , 49 , 50 ] , Inpainting [ 7 , 61 ] , Burst Image Deblurring [ 1 , 14 ] , and etc . These domains have achieved milestones in the 2D image and video domain [ 30 ] . However, the study on the restoration in 3D is insufficient. The restoration consistency of the texture and geometry from different views is hard to maintain when independently restoring them. 3D Reconstruction from Corrupted Inputs. 3D Gaussian Splatting (3DGS) [ 22 , 57 , 35 , 66 ] transforms input images into explicit-represented 3D Gaussian blobs with continuous optimization to reconstruct the large 3D scene. However, these models heavily rely on the condition of the input images. Motion blurring [ 53 , 70 ] or low-resolution [ 41 , 12 ] inputs that lack high-frequency and sharp details may greatly deteriorate the reconstruction quality or even fail at the initialization stage. To resolve this, methods like Bad-Gaussian [ 41 ] and SuperGaussian [ 70 ] depend on dense view input to iteratively optimize the degraded artifacts. Further, under heavily deteriorated inputs, these methods cannot recover high-quality outputs when they lack a strong generative prior [ 41 ] such as a pre-trained Diffusion Model. 3 Method Figure 2 : Method Overview. Our model is a UNet-based latent diffusion model that takes as input a degraded sparse image set and outputs a set of restored consistent images. The

Read the complete text