InstantSwap : Fast Customized Concept Swapping across Sharp Shape Differences

Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, Xiu Li · 2024

Read InstantSwap : Fast Customized Concept Swapping across Sharp Shape Differences by Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, Xiu Li (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01197.txt.

Introduction

Chenyang Zhu and seven co-authors present, in 2024 work, a fast tool that swaps an object in a photo for a custom one. Text-to-image models can do such swaps, but they often warp the rest of the scene, above all when the new object has a very unlike shape. They can also be slow to run or train.

Their tool, InstantSwap, first finds the object's box from the model's own focus maps. Inside the box, a cross-focus step feeds in the sense of both old and new objects. Outside it, changes are blocked so the backdrop stays put. To cut cost, slopes are worked out only at set steps, not each one. A new test set backs wide checks, and the team reports top results.

Catalog record

CREATORChenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, Xiu Li
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01197

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InstantSwap : Fast Customized Concept Swapping across Sharp Shape Differences Chenyang Zhu, Kai Li, Yue Ma, Longxiang Tang, Chengyu Fang, Chubin Chen, Qifeng Chen, Xiu Li Abstract Recent advances in Customized Concept Swapping (CCS) enable a text-to-image model to swap a concept in the source image with a customized target concept. However, the existing methods still face the challenges of inconsistency and inefficiency . They struggle to maintain consistency in both the foreground and background during concept swapping, especially when the shape difference is large between objects. Additionally, they either require time-consuming training processes or involve redundant calculations during inference. To tackle these issues, we introduce InstantSwap , a new CCS method that aims to handle sharp shape disparity at speed. Specifically, we first extract the bbox of the object in the source image automatically based on attention map analysis and leverage the bbox to achieve both foreground and background consistency. For background consistency, we remove the gradient outside the bbox during the swapping process so that the background is free from being modified. For foreground consistency, we employ a cross-attention mechanism to inject semantic information into both source and target concepts inside the box. This helps learn semantic-enhanced representations that encourage the swapping process to focus on the foreground objects. To improve swapping speed, we avoid computing gradients at each timestep but instead calculate them periodically to reduce the number of forward passes, which improves efficiency a lot with a little sacrifice on performance. Finally, we establish a benchmark dataset to facilitate comprehensive evaluation. Extensive evaluations demonstrate the superiority and versatility of InstantSwap . Fig 1 : Visual results of InstantSwap . Our approach can seamlessly swap a source concept with a customized concept in an image, even with great shape differences. Moreover, InstantSwap can be used for other tasks, such as concept insertion and removal. † † footnotetext: * Equal contribution. † † footnotetext: \dagger Corresponding Authors. 1 Introduction Fig 2 : Our InstantSwap achieves better swapping consistency than the existing methods. We explore the task of Customized Concept Swapping (CCS), a subtask of text-to-image (T2I) generation, which aims to replace a concept in a source image with a highly customized new concept. Combined with diffusion models ( Dhariwal and Nichol 2021 ; Rombach et al. 2022 ; Nichol et al. 2021 ) , recent CCS methods demonstrate widespread applicability in areas such as selfie enhancement, photo blog creation, and comic creation. Early work ( Yang et al. 2023 ) in CCS primarily relies on copy-paste techniques, which are rough and unreliable. By integrating powerful customization techniques ( Ruiz et al. 2023 ; Kumari et al. 2023 ) with image editing methods ( Hertz et al. 2022 ; Hertz et al. 2023 ) , a series of works ( Gu et al. 2024a ; Choi et al. 2023 ; Li et al. 2023 ; Gu et al. 2024b ) have been proposed. Although achieving remarkable success, these approaches still face the problems of inconsistency and inefficiency as shown in Fig. 2 . (1) Inconsistency : Attention-based methods such as PhotoSwap ( Gu et al. 2024a ) and P2P ( Hertz et al. 2022 ) maintain background consistency well but struggle with shape differences between source and target concepts, resulting in foreground inconsistency. Score distillation based methods such as SDS ( Poole et al. 2022 ) , DDS ( Hertz et al. 2023 ) , and CDS ( Nam et al. 2024 ) fail to generate foreground concepts precisely and alter the background significantly, causing both foreground and background inconsistency. (2) Inefficiency : Attention-based methods require an inefficient training phase ( Mokady et al. 2023 ; Ju et al. 2024 ) on source image to maintain background consistency. While score distillation based methods are training-free, they still require redundant calculations of forward passes at each timestep, leading to inference inefficiency. To address the aforementioned issues, we propose InstantSwap , a training-free framework that efficiently performs customized concept swapping across shape differences while maintaining both foreground and background consistency . Specifically, we extract the bounding box (bbox) that indicates the position of the source concept from the enhanced cross-attention map of the source image. With this bbox, we perform the background gradient masking (BGM) strategy to prevent modifications outside the bbox, thus ensuring background consistency. Moreover, to improve foreground consistency, we leverage the semantic information to highlight the cross-attention maps of source and target concepts respectively within the bbox. This strategy leads to the semantic-enhanced concept representation (SECR), which facilitates precise foreground swapping. Finally, we introduce the Step-skipping Gradient Updating (SSGU) strategy, which only performs forward passes at certain timesteps to calculate gradients. For the timesteps without direct gradient computations, we reuse the previously obtained gradients for updates. Through this strategy, we reduce the total number of forward passes and improve the efficiency of our method. Since the CCS is a recently proposed task, no dedicated evaluation benchmark currently exists. To address this gap, we introduce ConSwapBench , the first benchmark dataset specifically designed for CCS. ConSwapBench comprises two sub-benchmarks: ConceptBench and SwapBench. ConceptBench contains images representing target concepts, while SwapBench includes images with one or more concepts to be swapped, serving as source images. Through extensive qualitative and quantitative comparisons, we demonstrate the effectiveness and superiority of our InstantSwap . We also conduct comprehensive ablation studies to verify the effectiveness of each component of our approach. Additionally, we further extend our InstantSwap to related tasks, proving its efficacy and versatility. Our contributions are summarized as follows: • We propose InstantSwap , a novel training-free customized concept swapping (CCS) framework, which enables efficient concept swapping across sharp shape differences. • We design the background gradient masking (BGM) strategy and semantic-enhanced concept representation (SECR) to improve the background and foreground consistency respectively. Moreover, we adopt a step-skipping gradient updating (SSGU) strategy to reduce redundant computation and improve efficiency. • To provide a comprehensive evaluation for CCS, we introduce ConSwapBench , the first benchmark for customized concept swapping. Extensive qualitative and quantitative evaluations demonstrate the effectiveness and superiority of our InstantSwap . Fig 3 : Overall pipeline of InstantSwap . We first obtain the bbox of the source concept automatically. The obtained bbox is input into SECR in both the source and target branches to enhance the foreground swapping consistency. Additionally, the source and target branches generate the prediction of noise for the source and target images based on their respective prompts. The predicted noise, along with the bbox, is used for the BGM to preserve background consistency. 2 Related work 2.1 Diffusion-based Image Editing Image Editing is a fundamental and popular topic in computer vision. Previous works based on Generative Adversarial Networks (GAN) ( Goodfellow et al. 2020 ) only focus on specific object domains, which limits the application. With the emergence of diffusion model Rombach et al. 2022 , image editing is now able to modify various objects through prompts. These methods are mainly divided into five categories: instruction-based methods, blending-based, attention-based, inversion-based, and score distillation based methods. Instruction-based methods ( Brooks et al. 2023 ; Geng et al. 2024 ; Huang et al. 2024 ; Zhang et al. 2024a ; Ma et al. 2024a ) typically require an instruction

Read the complete text