PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control
Ruichen Wang, Junliang Zhang, Qingsong Xie, Chen Chen, Haonan Lu · 2024
Read PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control by Ruichen Wang, Junliang Zhang, Qingsong Xie, Chen Chen, Haonan Lu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01223.txt.
Introduction
Ruichen Wang and co-authors wrote this 2024 paper on a plug-in that steadies diffusion models when they fill in masked parts of pictures. Diffusion inpainting looks gorgeous but often drifts from the text prompt inside the masked patch and ignores how users actually draw masks. PainterNet slots into existing models to fix both habits.
Local prompt input lets users describe the patch itself, while attention control points and a new attention loss pin the focus of the model onto the right words and places. The mask generator was rebuilt to imitate real user scribbles, and the team released a matching training set, PainterData, plus a benchmark, PainterBench. Across quality and text-consistency metrics, local and global, the plug-in beat current state-of-the-art inpainting systems.
Catalog record
| CREATOR | Ruichen Wang, Junliang Zhang, Qingsong Xie, Chen Chen, Haonan Lu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01223
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PainterNet: Adaptive Image Inpainting with Actual-Token Attention and Diverse Mask Control Ruichen Wang, Junliang Zhang, Qingsong Xie, Chen Chen, Haonan Lu Abstract Recently, diffusion models have exhibited superior performance in the area of image inpainting. Inpainting methods based on diffusion models can usually generate realistic, high-quality image content for masked areas. However, due to the limitations of diffusion models, existing methods typically encounter problems in terms of semantic consistency between images and text, and the editing habits of users. To address these issues, we present PainterNet, a plugin that can be flexibly embedded into various diffusion models. To generate image content in the masked areas that highly aligns with the user input prompt, we proposed local prompt input, Attention Control Points (ACP), and Actual-Token Attention Loss (ATAL) to enhance the model’s focus on local areas. Additionally, we redesigned the MASK generation algorithm in training and testing dataset to simulate the user’s habit of applying MASK, and introduced a customized new training dataset, PainterData, and a benchmark dataset, PainterBench. Our extensive experimental analysis exhibits that PainterNet surpasses existing state-of-the-art models in key metrics including image quality and global/local text consistency. 1 Introduction The rapid progress of diffusion models Ho et al. 2020 ; Ho and Salimans 2022 has significantly advanced image generation techniques Chen et al. 2023a ; Saharia et al. 2022 ; Ramesh et al. 2022 , which are utilized in various applications, including prompt-based conditional editing Brooks et al. 2023 ; Hertz et al. 2022 , controllable generation Mou et al. 2024 ; Zhang et al. 2023b , and personalized image synthesis Lu et al. 2023 ; Ruiz et al. 2023 ; Gal et al. 2022 . Among these, image inpainting Xu et al. 2023b is a key application that uses guidance information to restore missing regions in images, allowing users to create content in specified areas based on textual prompts, making it a highly sought-after feature in recent years. Conventional diffusion-based inpainting methods can be divided into two main categories: modifying the sampling strategy Avrahami et al. 2023 ; Avrahami et al. 2022 ; Corneanu et al. 2024 ; Lugmayr et al. 2022 ; Zhang et al. 2023a and utilizing dedicated inpainting models Rombach et al. 2022 ; Wang et al. 2023 ; Xie et al. 2023a ; Yang et al. 2023 . The former involves sampling from a pretrained diffusion model over masked regions while maintaining the integrity of unmasked areas, which is a training-free approach. However, this method often leads to discontinuities in image generation due to inadequate attention to mask boundaries and contextual information. The latter approach enhances the diffusion model by extending input channels and fine-tuning to effectively address corrupted images and masks (as shown in Fig. 1 (a)). While dedicated inpainting models yield improved results, they require fine-tuning of the diffusion backbone and handle both conditioning and generation within a single UNet branch. This not only necessitates extensive data but also limits the model’s portability. Figure 1 : Comparison of previous inpainting architectures and PainterNet. (a) Dedicated Inpainting model based on diffusion model, enhanced by extended input channels and fine tuning. (b) We propose the plug-and-play approach PainterNet, which introduces an additional branch guidance model for hierarchical dense control via layer and attention control points. Figure 2 : Comparison of generation and datasets under local and global textual prompts. (a) Our method is able to generate correct results based on the local textual prompt, whereas other methods are not able to ensure consistency of generation when using global textual prompt. (b) In contrast to BrushData, our PainterData contains multiple types of masks (e.g., bounding box, irregular, segmentation-based) as well as local textual prompts generated by a multimodal large language models (MLLMs). Recently, control-based image colorization techniques, such as ControlNet-Inpainting Zhang et al. 2023b and BrushNet Ju et al. 2024 , have emerged as promising alternatives. These methods incorporate additional control branches to achieve flexible, plug-and-play solutions, and can perform well without the need for heavy training of dedicated inpainting models. Despite the significant improvements and flexibility offered by existing control-based methods, they still face several challenges. ControlNet-Inpainting Zhang et al. 2023b has flaws in terms of image pixel control, making it difficult for inpainted images to remain completely consistent with the original input. BrushNet Ju et al. 2024 is trained on segmentation mask data(as shown in the first row of Fig. 2 (b)), which introduces additional information(mask shape) during the training process while more flexible and personalized mask inputs are expected considering user habit. In addition, all the above methods usually rely on global textual prompts that do not provide localized detail descriptions, which may lead to inconsistencies between the generated local content and the expected prompts (as shown in Fig. 2 (a)). To address the limitations of existing methods, we propose a novel image editing framework—PainterNet. Specifically, inspired by BrushNet Ju et al. 2024 , we introduced an additional branch that employs a layered integration approach, progressively incorporating full UNet features into a pre-trained UNet, enabling dense pixel-wise control (as shown in Fig. 1 (b)). However, in order to obtain more detailed descriptions of the mask area from the input prompt, we adopted local textual prompts as the input. Further more, we introduced Attention Control Points (ACP) and Actual-Token Attention Loss (ATAL) to make the model focus more on the mask area based on the information provided by the input local textual prompts. We have also constructed a new training dataset, PainterData, and a corresponding benchmark, PainterBench, based on BrushData Ju et al. 2024 . Considering the actual applications of users, We designed a diversified mask generation strategy to fit the types of masks that may appear in the actual use of the inpainting model (including boundary box and irregular scribbles to simulate fingers, as shown in the second row of Fig. 2 (b)). In addition, we leveraged multimodal large language models (MLLMs), such as ShareGPT Chen et al. 2023b , to generate local textual prompts for the masked regions, encouraging the model to focus on generating localized content. The contributions of our work are summarized as follows: \bullet We proposed a novel plug-and-play image inpainting framework, PainterNet, which introduces an additional branch for layered and dense pixel-wise control, enhancing the generation capabilities of the diffusion model. \bullet We introduced Attention Control Points (ACP) and the Actual-Token Attention Loss (ATAL) to capture the semantic associations between masked images and local textual prompts. This ensures that our model can utilize the information in the local textual prompt well to generate the missing area in the image. \bullet We constructed a new dataset pipeline, PainterData, which automatically generates local textual prompts using multimodal large language models, and design diverse mask generation strategy. This enables the model to better understand the semantic information of local regions, making it more applicable to real-world scenarios. Figure 3 : Overview of our method. Our PainterNet introduces an additional branch that uses a hierarchical approach to gradually incorporate the complete UNet features into the pre-trained UNet layer by layer through layers and attentional control points. Meanwhile, we designed the Actual-Token Attention Loss (ATAL) \mathcal{L}_{\text{ATAL}} to direct the model’s attention to the mask region. Masking Strategy generates diverse masks (e.g.,