SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion
Trong-Tung Nguyen, Quang Nguyen, Khoi Nguyen, Anh Tran, Cuong Pham †, Qualcomm AI Research · 2024
Read SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion by Trong-Tung Nguyen, Quang Nguyen, Khoi Nguyen, Anh Tran, Cuong Pham †, Qualcomm AI Research (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04301.txt.
Introduction
Trong-Tung Nguyen and co-authors present SwiftEdit, a tool that edits images from text instructions in a fraction of a second. Their 2024 paper notes that text-guided editing has grown powerful through diffusion models. Those models, though, need many costly steps to invert and resample an image, which is too slow for phones and other on-device uses.
SwiftEdit rebuilds the image in a single inversion step instead of many. A mask-guided method with an attention adjustment then applies the edit only where wanted. The tool finishes an edit in about a quarter of a second, at least fifty times faster than earlier methods. Quality stayed competitive in their tests.
Catalog record
| CREATOR | Trong-Tung Nguyen, Quang Nguyen, Khoi Nguyen, Anh Tran, Cuong Pham †, Qualcomm AI Research |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04301
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SwiftEdit: Lightning Fast Text-Guided Image Editing via One-Step Diffusion Trong-Tung Nguyen, Quang Nguyen, Khoi Nguyen, Anh Tran, Cuong Pham †, Qualcomm AI Research Abstract Recent advances in text-guided image editing enable users to perform image edits through simple text inputs, leveraging the extensive priors of multi-step diffusion-based text-to-image models. However, these methods often fall short of the speed demands required for real-world and on-device applications due to the costly multi-step inversion and sampling process involved. In response to this, we introduce SwiftEdit, a simple yet highly efficient editing tool that achieve instant text-guided image editing ( in 0.23s ). The advancement of SwiftEdit lies in its two novel contributions: a one-step inversion framework that enables one-step image reconstruction via inversion and a mask-guided editing technique with our proposed attention rescaling mechanism to perform localized image editing. Extensive experiments are provided to demonstrate the effectiveness and efficiency of SwiftEdit. In particular, SwiftEdit enables instant text-guided image editing, which is extremely faster than previous multi-step methods (at least 50 × \times times faster ) while maintain a competitive performance in editing results. Our project is at https://swift-edit.github.io/ . Figure 1 : SwiftEdit empowers instant, localized image editing using only text prompts, freeing users from the need to define masks. In just 0.23 seconds on a single A100 GPU, it unlocks a world of creative possibilities demonstrated across diverse editing scenarios. † † + Qualcomm Vietnam Company Limited † † † also affiliated with Posts & Telecom. Inst. of Tech., Vietnam † † Contact email: [email protected] 1 Introduction Recent text-to-image diffusion models [ 27 , 26 , 24 , 5 ] have achieved remarkable results in generating high-quality images semantically aligned with given text prompts. To generate realistic images, most of them rely on multi-step sampling techniques, which reverse the diffusion process starting from random noise to realistic image. To overcome this time-consuming sampling process, some works focus on reducing the number of sampling steps to a few (4-8 steps) [ 29 ] or even one step [ 40 , 39 , 20 , 5 ] via distillation techniques while not compromising results. These approaches not only accelerate image generation but also enable faster inference for downstream tasks, such as image editing. For text-guided image editing, recent approaches [ 19 , 11 , 13 ] use an inversion process to determine the initial noise for a source image, allowing for (1) source image reconstruction and (2) content modification aligned with guided text while preserving other details. Starting from this inverted noise, additional techniques, such as attention manipulation and hijacking [ 3 , 35 , 21 ] , are applied at each denoising step to inject edits gradually while preserving key background elements. This typical approach, however, is resource-intensive, requiring two lengthy multi-step processes: inversion and editing. To address this, recent works [ 6 , 8 , 33 ] use few-step diffusion models, like SD-Turbo [ 30 ] , to reduce the sampling steps required for inversion and editing, incorporating additional guidance for disentangled editing via text prompts. However, these methods still struggle to achieve sufficiently fast text-guided image editing for on-device applications while maintaining performance competitive with multistep approaches. Figure 2 : Comparing our one-step SwiftEdit with few-step and multi-step diffusion editing methods in terms of background preservation (PSNR), editing semantics (CLIP score), and runtime. Our method delivers lightning-fast text-guided editing while achieving competitive results. In this work, we take a different approach by building on a one-step text-to-image model for image editing. We introduce SwiftEdit – the first one-step text-guided image editing tool – which achieves at least 50 \times faster execution than previous multi-step methods while maintaining competitive editing quality. Notably, both the inversion and editing in SwiftEdit are accomplished in a single step. Inverting one-step diffusion models is challenging, as existing techniques like DDIM Inversion [ 31 ] and Null-text Inversion [ 19 ] are unsuitable for our one-step real-time editing goal. To achieve this, we design a novel one-step inversion framework inspired by encoder-based GAN Inversion methods [ 41 , 36 , 37 ] . Unlike GAN inversion, which requires domain-specific networks and retraining, our inversion framework generalizes to any input images. For this, we leverage SwiftBrushv2 [ 5 ] , a recent one-step text-to-image model known for speed, diversity, and quality, using it as both the one-step image generator and backbone for our one-step inversion network . We then train it with weights initialized from SwiftBrushv2 to handle any source inputs through a two-stage training strategy, combining supervision from both synthetic and real data. Following the one-step inversion, we introduce an efficient mask-based editing technique. Our method can either accept an input editing mask or infer it directly from the trained inversion network and guidance prompts. The mask is then used in our novel attention-rescaling technique to blend and control the edit strength while preserving background elements, enabling high-quality editing results. To the best of our knowledge, our work is the first to explore diffusion-based one-step inversion using a one-step text-to-image generation model to instantly perform text-guided image editing ( in 0.23 seconds ). While being significantly fast compared to other multi-step and few-step editing methods, our approach achieves a competitive editing result as shown in Fig. 2 . In summary, our main contribution includes: • We propose a novel one-step inversion framework trained with a two-stage strategy. Once trained, our framework can invert any input images into an editable latent in a single step without further retraining or finetuning. • We show that our well-trained inversion framework can produce an editing mask guided by source and target text prompts within a single batchified forward pass. • We propose a novel attention-rescaling technique for mask-based editing, offering flexible control over editing strength while preserving key background information. 2 Related Work 2.1 Text-to-image Diffusion Models Diffusion-based text-to-image models [ 24 , 27 , 26 ] typically rely on computationally expensive iterative denoising to generate realistic images from Gaussian noise. Recent advances [ 28 , 18 , 32 , 16 ] alleviate this by distilling the knowledge from multi-step teacher models into a few-step student network. Notable works [ 15 , 32 , 16 , 40 , 39 , 20 , 5 ] show that this knowledge can be distilled even into a one-step student model. Specifically, Instaflow [ 15 ] uses rectified flow to train a one-step network, while DMD [ 40 ] applies distribution-matching objectives for knowledge transfer. DMDv2 [ 39 ] removes costly regression losses, enabling efficient few-step sampling. SwiftBrush [ 20 ] utilizes an image-free distillation method with text-to-3D generation objectives, and SwiftBrushv2 [ 5 ] integrates post-training model merging and clamped CLIP loss, surpassing its teacher model to achieve state-of-the-art one-step text-to-image performance. These one-step models provide rich prior information about text-image alignment and are extremely fast, making them ideal for our one-step text-based image editing approach. 2.2 Text-based Image Editing Several approaches leverage the strong prior of image-text relationships in text-to-image models to perform text-guided multi-step image editing via an inverse-to-edit approach. First, they invert the source image into “informative” noise. Methods like DDIM Inversion [ 31 ] use linear approximations of noise prediction, while Null-text Inversion [ 19 ] enhances reconstruction quality through