CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing
Xiaole Xian, Xilin He, Zenghao Niu, Junliang Zhang, Weicheng Xie, Siyang Song, Zitong Yu, Linlin Shen · 2024
Read CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing by Xiaole Xian, Xilin He, Zenghao Niu, Junliang Zhang, Weicheng Xie, Siyang Song, Zitong Yu, Linlin Shen (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-13565.txt.
Introduction
Xiaole Xian and co-authors present a method for editing face photos in a 2024 paper. The goal is to change one local trait, such as skin or hair, while leaving the rest of the face alone. Many older tools need fresh tuning for each effect, or they spill changes past the target area.
Their method, CA-Edit, works like inpainting, filling in just the chosen region. They build training sets that link a trait phrase, a text prompt, and an image. An adapter then carries fine skin detail from the source photo, while a guidance step keeps tones smooth at the border. The aim is to stop clashes between the photo cues and the text cues. Tests, both scored and visual, show gains in faithfulness and in how well the edit can be steered.
Catalog record
| CREATOR | Xiaole Xian, Xilin He, Zenghao Niu, Junliang Zhang, Weicheng Xie, Siyang Song, Zitong Yu, Linlin Shen |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-13565
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing Xiaole Xian, Xilin He, Zenghao Niu, Junliang Zhang, Weicheng Xie, Siyang Song, Zitong Yu, Linlin Shen Abstract For efficient and high-fidelity local facial attribute editing, most existing editing methods either require additional fine-tuning for different editing effects or tend to affect beyond the editing regions. Alternatively, inpainting methods can edit the target image region while preserving external areas. However, current inpainting methods still suffer from the generation misalignment with facial attributes description and the loss of facial skin details. To address these challenges, (i) a novel data utilization strategy is introduced to construct datasets consisting of attribute-text-image triples from a data-driven perspective, (ii) a Causality-Aware Condition Adapter is proposed to enhance the contextual causality modeling of specific details, which encodes the skin details from the original image while preventing conflicts between these cues and textual conditions. In addition, a Skin Transition Frequency Guidance technique is introduced for the local modeling of contextual causality via sampling guidance driven by low-frequency alignment. Extensive quantitative and qualitative experiments demonstrate the effectiveness of our method in boosting both fidelity and editability for localized attribute editing. The code is available at https://github.com/connorxian/CA-Edit . Introduction Efficient and high-fidelity local facial attribute editing with textual description represents a challenging task in computer vision. GANs-based methods ( Wang et al. 2022 ; Pernuš, Štruc, and Dobrišek 2023 ) have explored this task, which primarily optimize the original image within the latent space with a pre-trained StyleGAN model ( Karras et al. 2020 ) . However, these GANs-based methods require additional fine-tuning for different attributes. Subsequently, the prior diffusion-based image editing methods based on the text-to-image (T2I) diffusion models achieve image editing in various ways. These methods are either based on P2P ( Hertz et al. 2022 ) , utilizing the original image attention injection mechanism to preserve the layout, or based on DDIM Inversion ( Song, Meng, and Ermon 2020 ) , modifying the latent at the noise level. However, such methods may lead to inconsistencies beyond the editing target area. Figure 1: (Top) The existing text-guided inpainting pipeline for our local attribute editing task. (Bottom) Our method takes account of the causality of the the specific details from the original image, improving the editability and the fidelity. Regarding local facial attribute editing, image inpainting is a technique focused on local masked region painting, which also benefits from the recent advances in diffusion models ( Avrahami, Lischinski, and Fried 2022 ; Yang et al. 2023 ; Yang, Chen, and Liao 2023 ) . Besides, image inpainting has been also developed for local facial attribute editing, which focuses on the inpainting of local masked regions, based on advanced diffusion models ( Avrahami, Lischinski, and Fried 2022 ; Yang et al. 2023 ; Yang, Chen, and Liao 2023 ) . Text-guided image inpainting ( Avrahami, Lischinski, and Fried 2022 ) allows prompt-driven content generation in specific areas without finetuning during inference, while maintaining consistency between the editing and unmasked regions, which is thus used in our method. However, existing methods for image inpainting may suffer from concerns in terms of editability and fidelity. The first problem: they ( Zhang, Rao, and Agrawala 2023 ; Ju et al. 2024 ) struggle to understand the contextual relationship between unmasked facial regions and the textual description, resulting in the neglect of the text prompt while creating a plain completion ( Fig. 1 (a) ). For addressing this problem, Hd-Painter ( Manukyan et al. 2023 ) can better align the inpainting generation with the text by modifying the latent, while it still fails for local facial text prompts. The root cause is that previous diffusion models are primarily trained on natural image-text pairs, lacking the fine-grained knowledge of human faces. The second problem: For facial inpainting, previous works ( Rombach et al. 2022 ; Yang, Chen, and Liao 2023 ) do not take adequate consideration of the contextual causality between the masked region and the specific details (skin texture, skin tone, and other details) of the original image. The causality consideration is further constrained by the conflict between textual editing conditions and the preservation of these details in original image. In facial images, even slight differences in these details become visibly obvious, largely impairing the overall naturalness. ( Fig. 1 (b) ) Therefore, the key to maintaining the skin details and mitigating the difference lies in the reasonably causality-aware modeling of these specific details from the original image. For addressing this problem, existing approaches adapt the parallel attention with textual conditions(i.e. IP-Adapter ( Ye et al. 2023 ) ) to inject original image information and enhance contextual causality modeling. However, as shown in Fig. 1 , this causality conflicts modeling with the text condition may lead to severe content leakage. ( Fig. 1 (c) ) Meanwhile, from a localized contextual perspective, existing methods ( Ju et al. 2024 ; Xu et al. 2024 ) lack explicit approaches for this fine-grained local context, causing disharmony in boundary regions of the primary editing regions, while the skin transitions are generally smooth. To address these challenges, we proposed our CA-Edit from the local attribute data construction and causality-aware condition adapter. For addressing the first problem , training on detailed textual captions of local facial attributes would be crucial for editability. To this end, we introduce a data construction pipeline, leveraging Multimodal Large Language Models (MLLMs) ( Chen et al. 2023 ; Li et al. 2023 ) for automatic local facial attribute captioning and the face parsing model for segmentation acquisition. For addressing the second problem , we introduce an additional adapter for original image condition, as well as a sampling guidance during inference, to fully explore original image cues. Specifically, (i) the Causality-Aware Condition Adapter ( \text{CA}^{2} ) is proposed to enhance the causality modeling while preventing the conflict with textual condition. (ii) a sampling guidance technique called Skin Transition Frequency Guidance (STFG) is proposed to mitigate the artifacts on the ‘boundary regions’ via enhancing the similarity between the generated image and the low-frequency components of the original image. The main contributions of this work are summarized as: • To address the limitations of existing datasets lacking local facial attribute captions, we propose LAMask-Caption, the first dataset with detailed local facial captions which contains 200,000 high-quality facial images and employs Large Multimodal Models (MLMMs) for automatic captioning of local facial regions. • To jointly address the issues of fine-grained context modeling and content leakage, we propose the novel \text{CA}^{2} ) that enhances contextual causality modeling in primary editing regions while regularizing the visual condition according to the textual condition and latent. Furthermore, we propose the novel STFG to preserve the skin details on the boundary regions by enhancing the low-frequency similarity with the original image during inference. • Quantitative and qualitative experiments demonstrate that CA-Edit produces more harmonious and natural outcomes, showcasing the superiority of our method in local attribute editing. Related Work Generative Face Editing The advancement of facial editing and manipulation has been promoted by the emergence of recent generative approaches. Early efforts in this area have explored the application of GANs-based models ( Karras, Laine, and Aila 2019 ; Shen et al. 2020 ;