Detail-Preserving Latent Diffusion for Stable Shadow Removal

Jiamin Xu, Yuxin Zheng, Zelong Li, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu · 2024

Read Detail-Preserving Latent Diffusion for Stable Shadow Removal by Jiamin Xu, Yuxin Zheng, Zelong Li, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17630.txt.

Introduction

Jiamin Xu and co-authors erase shadows from photos using a diffusion model that refuses to blur the details. Shadow removal often fails on new scenes because training sets are small and similar. The team borrows the visual knowledge of a pre-trained Stable Diffusion model and adapts it in two stages.

First they tune the model's denoiser while freezing its image codec, which lifts shadows but can soften fine texture. A second stage then pipes detail features from the encoder into the decoder, restoring sharpness where it was lost. On benchmarks the method beats prior state of the art, and on datasets it never trained on it still holds up, a sign the borrowed knowledge travels well.

Catalog record

CREATORJiamin Xu, Yuxin Zheng, Zelong Li, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17630

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Detail-Preserving Latent Diffusion for Stable Shadow Removal Jiamin Xu, Yuxin Zheng, Zelong Li, Chi Wang, Renshu Gu, Weiwei Xu, Gang Xu Abstract Achieving high-quality shadow removal with strong generalizability is challenging in scenes with complex global illumination. Due to the limited diversity in shadow removal datasets, current methods are prone to overfitting training data, often leading to reduced performance on unseen cases. To address this, we leverage the rich visual priors of a pre-trained Stable Diffusion (SD) model and propose a two-stage fine-tuning pipeline to adapt the SD model for stable and efficient shadow removal. In the first stage, we fix the VAE and fine-tune the denoiser in latent space, which yields substantial shadow removal but may lose some high-frequency details. To resolve this, we introduce a second stage, called the detail injection stage. This stage selectively extracts features from the VAE encoder to modulate the decoder, injecting fine details into the final results. Experimental results show that our method outperforms state-of-the-art shadow removal techniques. The cross-dataset evaluation further demonstrates that our method generalizes effectively to unseen data, enhancing the applicability of shadow removal methods. 1 Introduction Shadows are natural phenomena that occur when an obstacle blocks global light transport. They provide crucial cues for interpreting the three-dimensional structure of objects and scenes [ 44 ] . However, shadows can also hinder computer vision tasks, such as object segmentation [ 8 ] , tracking [ 32 ] , and intrinsic decomposition [ 16 , 17 , 25 , 43 ] . In recent years, deep-learning-based methods for shadow removal [ 4 , 40 , 5 , 24 ] have shown significant advancements by learning pixel-wise mappings between shadowed images and their corresponding ground-truth shadow-free versions in a fully supervised manner. However, such approaches often risk overfitting the limited training datasets [ 14 , 28 , 35 ] , leading to inadequate generalization. Figure 1 : Top : For complex shadows in indoor scenes, current methods struggle to completely remove shadows, such as those behind the eagle sculpture. Bottom : Our method effectively removes the shadows, and with our second stage, the details from the original input, such as the wooden texture shown in the cropped image, are preserved. Drawing inspiration from recent advancements in zero-shot dense prediction [ 12 , 42 ] , we leverage the robust and comprehensive visual priors of the pre-trained Stable Diffusion (SD) model [ 29 ] , which has been trained on billions of images, to enhance the generalization capability of shadow removal. However, developing a fine-tuning strategy that effectively removes shadows while preserving clear details in both shadowed and non-shadowed areas is not trivial. Unlike low-frequency depth and normal prediction tasks [ 12 , 42 ] , applying the SD model to shadow removal can result in detail loss and misalignment between the conditional image and the shadow-free output (Fig. 1 ). It is because the SD model, which consists of a Variational Autoencoder (VAE) and a Latent Diffusion Model (LDM), is designed for text-to-image generation and can lead to lossy compression during the mapping from pixel space ( W\times H\times 3 ) to latent space ( \frac{W}{8}\times\frac{H}{8}\times 4 ). To achieve stable and detail-preserving shadow removal using the pre-trained SD priors, we introduce a two-stage pipeline: 1) eliminating shadows in latent space by fixing the pre-trained VAE and fine-tuning the LDM and 2) injecting shadow-aware multi-resolution details from the original input into the VAE decoding process. In the first stage, our method retains the pre-trained VAE and fine-tunes the LDM by conditioning it on the latent features of the input shadow image. We do not fine-tune the VAE, as it effectively handles shadow-free images despite not being specifically trained for this purpose, and the scale of our shadow-free images is limited. Our findings indicate that the latent space from the VAE, combined with the generative priors from the LDM, enhances shadow removal tasks, particularly when annotated training data is scarce. By operating in a low-resolution latent space rather than high-resolution pixel space, our method performs global self-attention with acceptable memory consumption, capturing more global contextual information compared to convolution and window-based self-attention [ 21 ] . The fine-tuned SD model in the first stage produces visually appealing results, but it may lose some local details. To further enhance the quality of shadow removal, we propose a second stage for injecting shadow-free details. In this stage, we introduce a Detail Injection (DI) module that enhances the decoded latent features from the pre-trained VAE decoder by incorporating fine details from the input images. The VAE parameters are kept fixed. To prevent shadows from being reintroduced as “details”, our DI module is equipped with shadow perception capabilities. Inspired by the observation that the shadow mask in the current shadow removal dataset is derived from the binary difference between shadowed and shadow-free images, we utilize concatenated features from the input shadow images and the coarse, shadow-free images decoded from the LDM output as cues for shadow-free detail injection. By combining features from both image types, the DI module can implicitly detect shadows and learn to inject shadow-free details into the VAE decoders. Note that Refusion [ 23 ] also performs latent-space diffusion for shadow removal. It captures input image information in the decoder [ 23 ] using a U-Net architecture specifically designed for large image shadow removal. However, the primary difference lies in the network design and fine-tuning strategy. Our method is designed to leverage the high-quality priors of the pre-trained VAE and LDM in the SD model, resulting in superior shadow removal quality. Our method can also handle high-resolution images (e.g., 1920×1440 in the WRSD dataset [ 35 ] ), providing greater shadow removal efficiency compared to patch-based diffusion methods [ 22 ] . Overall, our contributions can be summarized as follows. • We propose a two-stage fine-tuning pipeline to transform a pre-trained Stable Diffusion model into an image-conditional shadow-free image generator. This approach enables robust, high-resolution shadow removal without an input shadow mask. • We introduce a shadow-aware detail injection module that utilizes the VAE encoder features to modulate the pre-trained VAE decoders, selectively aligning per-pixel details from the input image with those in the output shadow-free image. • Extensive experimental results on public datasets demonstrate that the proposed method significantly outperforms state-of-the-art shadow removal methods. Additionally, we show that our method exhibits better generalizability by training on one dataset and testing on another. 2 Related Work 2.1 Deep Shadow Removal Figure 2 : Our proposed network. We propose a two-stage shadow removal network based on Stable Diffusion (SD). (1) In the first stage, as shown in the bottom half, we fine-tune the pre-trained UNet in SD within the latent space defined by SD’s pre-trained VAE ( \mathcal{E} and \mathcal{D} ). We found that the pre-trained latent space can effectively represent shadow-free images. (2) In the second stage, as shown in the top half, we modulate the VAE decoder \mathcal{D} by selectively adding features from the VAE encoder \mathcal{E} using a Detail Injection Model (DIM). The model consists of multiple RRDB layers, which inject shadow-free texture details into the decoder features. With these two stages, our proposed network can generate high-quality, shadow-free images that preserve

Read the complete text