From Missing Pieces to Masterpieces: Image Completion with Context-Adaptive Diffusion

Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Michael Felsberg, Dacheng Tao, Xuelong Li · 2025

Read From Missing Pieces to Masterpieces: Image Completion with Context-Adaptive Diffusion by Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Michael Felsberg, Dacheng Tao, Xuelong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14294.txt.

Introduction

Pourya Shamsolmoali and co-authors present a method for filling in missing parts of images so the patch blends with what remains, in this 2025 paper. Diffusion models can invent plausible content, but the invented part often clashes with the real surroundings in layout or meaning. The models lean on global habits rather than the local scene.

Their framework, ConFill, adds a check that runs during generation. At each step it measures the gap between the known and the filled regions and pulls them into line. A sampling scheme also spends more steps where the missing content is hardest to rebuild, sharpening detail there. In wide tests the method beat current rivals and set a new mark for the task. The authors aim at photo repair, restoration, and editing where seams cannot show.

Catalog record

CREATORPourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Michael Felsberg, Dacheng Tao, Xuelong Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14294

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Missing Pieces to Masterpieces: Image Completion with Context-Adaptive Diffusion Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou, Michael Felsberg, Dacheng Tao, Xuelong Li Abstract Image completion is a challenging task, particularly when ensuring that generated content seamlessly integrates with existing parts of an image. While recent diffusion models have shown promise, they often struggle with maintaining coherence between known and unknown (missing) regions. This issue arises from the lack of explicit spatial and semantic alignment during the diffusion process, resulting in content that does not smoothly integrate with the original image. Additionally, diffusion models typically rely on global learned distributions rather than localized features, leading to inconsistencies between the generated and existing image parts. In this work, we propose ConFill, a novel framework that introduces a Context-Adaptive Discrepancy (CAD) model to ensure that intermediate distributions of known and unknown regions are closely aligned throughout the diffusion process. By incorporating CAD, our model progressively reduces discrepancies between generated and original images at each diffusion step, leading to contextually aligned completion. Moreover, ConFill uses a new Dynamic Sampling mechanism that adaptively increases the sampling rate in regions with high reconstruction complexity. This approach enables precise adjustments, enhancing detail and integration in restored areas. Extensive experiments demonstrate that ConFill outperforms current methods, setting a new benchmark in image completion. I Introduction Image completion, also called image inpainting, is an important task in computer vision and image processing that aims to reconstruct missing or corrupted portions of an image such that the restored areas are both visually pleasing and coherent. This technique is widely used in various domains, including image editing [ 1 , 2 ] , image restoration [ 3 , 4 ] , and the removal of unwanted objects [ 5 ] . At present, several state-of-the-art methods use approaches such as GANs [ 6 , 7 , 8 ] , transformers [ 9 , 10 ] , and autoregressive modeling [ 11 , 12 ] . Despite their advancements, these techniques often experience unstable training [ 12 ] and struggle with filling large missing regions in images [ 8 , 2 ] . Recently, there has been a growing interest among researchers in using diffusion models [ 13 , 14 ] to address the above issue. These generative models transform noisy images into natural-looking images through sequential denoising steps. A common strategy in this domain is to use a pre-trained, static diffusion model, which eliminates the need for additional training and enhances the model’s flexibility and adaptability. Fig. 1 : (a) Generated images by DPS, RePaint, and our ConFill, using a fixed diffusion model on different masked inputs. (b) LPIPS score vs. denoising timesteps for a single input image. These methods address image completion constraints by using various replacement strategies. For example, by replacing predicted image regions with corresponding parts from the reference image [ 15 ] , or by incorporating corrupted regions from the reference image into the intermediate denoising process [ 16 , 17 ] . Similarly, [ 2 ] focuses on optimizing pixel distribution rather than denoising, aiming to reduce the number of iterations needed for training. Although these methods modify large portions of an image, they struggle to properly adjust the pixels in unknown areas to ensure consistency with the surrounding context, resulting in noticeable discontinuities, which significantly compromise the overall quality of the generated images [ 18 ] . Additionally, the sampling techniques used in [ 19 , 20 , 21 ] often compromise sample quality by introducing additional noise, further limiting their effectiveness. In image completion, the efficacy of diffusion models is significantly influenced by the discrepancy metrics they use [ 22 , 23 ] . These metrics are important as they determine the fidelity of the transition from noisy image to an output that aligns with the original image regions. Traditional discrepancy metrics used in diffusion models, such as Euclidean distance, generally focus on minimizing statistical disparities without considering the complex spatial and structural details of the images. For example, two images might have similar pixel values but different textures or structures, leading to high Euclidean distance but visually incoherent outputs [ 24 ] . Moreover, the limitations of traditional discrepancy metrics become more visible when models are required to adapt to the specific needs of different regions of an image. Indeed, each part of an image may have unique characteristics such as texture, lighting, or context that require specific handling to achieve a realistic completion. DPS [ 25 ] and SMCDiff [ 18 ] use Bayesian optimization for conditioned sampling, however, their reliance on computationally intensive processes and approximations compromises both accuracy and visual quality. Fig. 1 illustrates examples of incoherent image completions for masked images. DPS shows a noticeable disparity between the left and right sides of the generated images. RePaint also has discontinuities in the reconstructions. MCG [ 26 ] uses manifold constraints, but this can restrict diffusion process, leading to underfitting and failing to capture complex data variability. CoPaint [ 27 ] addresses this problem by modifying both the known and unknown regions to achieve more seamless integration. This approach alters key image features, such as edges that should remain intact, leading to errors that compromise the objectives of image completion. In these diffusion models, the focus is on learning the overall noise distribution, with limited attention to the alignment between consecutive denoising steps. While this assumption of treating each step as independent can be effective for completing small missing regions, it becomes problematic when dealing with large or complex areas. Without explicitly modeling the transitions between steps, inconsistencies can arise, leading to artifacts or unrealistic completions, particularly in regions requiring detailed reconstruction. A potential solution is to optimize the mapping between inverse diffusion steps. Recently, [ 23 ] uses Brenier theorem [ 28 , 29 ] to ensure structural fidelity during transformations between two probability distributions for generative task. Brenier theorem provides a globally optimal solution for mapping distributions, but it does not consider local coherence within patches of the image [ 30 ] . In diffusion-based image completion, maintaining smooth and coherent transitions across these local patches is essential. This gap highlights the need for a more flexible alignment process that can better handle the diverse patterns and textures within an image. In this paper, we introduce ConFill, a novel framework specially designed for image completion. ConFill overcomes the limitations of traditional discrepancy metrics in diffusion models and enhances both precision and contextual relevance in image reconstruction. At the core of ConFill is the Context-adaptive Discrepancy (CAD) method, which redefines the denoising process by minimizing inter-step discrepancies in the diffusion sequence, ensuring smoother transitions and improving alignment with the original image content. CAD incorporates a context-sensitive adaptation mechanism that dynamically adjusts to local image attributes throughout the diffusion process. This ensures that each image region is processed based on its unique characteristics, thereby enhancing the alignment of the reconstructed image with the original content. As a result, every step in the diffusion cycle is optimized for better contextual consistency. Moreover, ConFill enhances image reconstruction quality by selectively sampling from the posterior distribution of intermediate images, which helps reduce errors and improve the quality of the denoised output. Indeed, ConFill

Read the complete text