DreamOmni: Unified Image Generation and Editing

Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia · 2024

Read DreamOmni: Unified Image Generation and Editing by Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17098.txt.

Introduction

Bin Xia and co-authors build DreamOmni, a single model that both creates images from text and edits them in several ways. Large language models show that one system can serve many tasks better than separate tools. Image generators, though, were not first designed with editing in mind. Studying existing frameworks and editing needs, the team designed one framework covering generation plus instruction-based and drag-based editing.

A key part of the work is data: good editing examples are hard to make. The team built a pipeline using sticker-like elements to synthesise accurate training pairs cheaply and at scale. Training the two kinds of tasks together helped both sides. Generation improved the model's grasp of concepts, and editing sharpened its control. Tests confirm the unified model performs strongly across its tasks. The authors plan to release code and model for others to use.

Catalog record

CREATORBin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17098

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DreamOmni: Unified Image Generation and Editing Bin Xia, Yuechen Zhang, Jingyao Li, Chengyao Wang, Yitong Wang, Xinglong Wu, Bei Yu, Jiaya Jia Abstract Currently, the success of large language models (LLMs) illustrates that a unified multitasking approach can significantly enhance model usability, streamline deployment, and foster synergistic benefits across different tasks. However, in computer vision, while text-to-image (T2I) models have significantly improved generation quality through scaling up, their framework design did not initially consider how to unify with downstream tasks, such as various types of editing. To address this, we introduce DreamOmni, a unified model for image generation and editing. We begin by analyzing existing frameworks and the requirements of downstream tasks, proposing a unified framework that integrates both T2I models and various editing tasks. Furthermore, another key challenge is the efficient creation of high-quality editing data, particularly for instruction-based and drag-based editing. To this end, we develop a synthetic data pipeline using sticker-like elements to synthesize accurate, high-quality datasets efficiently, which enables editing data scaling up for unified model training. For training, DreamOmni jointly trains T2I generation and downstream tasks. T2I training enhances the model’s understanding of specific concepts and improves generation quality, while editing training helps the model grasp the nuances of the editing task. This collaboration significantly boosts editing performance. Extensive experiments confirm the effectiveness of DreamOmni. The code and model will be released. Figure 1 : The gallery of DreamOmni. DreamOmni, as a native unified image generation and editing model, can handle various tasks. 1 Introudction Recently, text-to-image (T2I) generative foundation models [ 44 , 39 , 15 , 47 , 6 , 27 ] have made remarkable progress, driving the development of various downstream applications such as image editing, video generation, and more. Currently, T2I foundation models face two significant challenges: (1) Adapting these models for downstream applications often requires the integration of various plugins (such as ControlNet [ 62 ] and IP-adapter [ 60 ] ) in different ways, or the extension of input channels ( e.g . , SD-inpainting [ 44 ] , InstructP2P [ 8 ] ). This reliance on specialized frameworks hampers multi-task generalization and complicates deployment. (2) High-quality and accurate editing data is difficult to obtain, including instruction-based editing, drag editing, and subject-driven generation data. In this paper, we propose to unify T2I models with multiple editing tasks, such as instruction-based editing, inpainting & outpainting, drag editing, and reference image generation within a single framework. Additionally, we introduce an efficient synthetic data pipeline for efficiently and accurately constructing editing data, facilitating the training of native unified generation and editing model. We began by exploring fundamental model frameworks to develop a unified framework that is both highly effective and efficient, achieving rapid training convergence. (1) We aligned certain frameworks, such as Unet [ 44 ] and DIT [ 38 ] , to have similar parameter sizes and runtime settings for fair comparison (Fig. 3 ). Our analysis showed that DIT’s effectiveness comes from concentrating most of its computational and parameter load on latent at 2 \times downsampling size, whereas Unet distributes more of its operations at smaller scales. This allocation makes DIT comparatively superior, as concentrating computations at the 2 \times downsampled latent achieves a better trade-off. However, we observed that Unet’s residual connections significantly improve the model’s training convergence speed. Inspired by the efficiency of high-resolution latent computations, we also incorporated additional residual convolutional networks for the input-scale latent. (2) We replaced the original text encoder with a vision-language model (VLM) to unify visual-language prompt encoding. This encoded prompt was then concatenated with the noisy latent, allowing for integrated computation within the DIT framework. Next, we need to obtain accurate training data efficiently for our unified model. For various types of editing, such as instruction-based editing, it’s challenging to create and filter pairs of data that precisely match the instructions while avoiding detail corruption. We have discovered that the key to effective editing lies in helping the model understand the meaning of the editing operations, rather than learning specific concepts (a capability the model already possesses in T2I training). Therefore, we propose a synthetic collage data pipeline that easily scales up to generate accurate and diverse edited data, enabling models to better understand the nuances of specific editing tasks (Fig. 2 (b)). (1) For instruction-based editing, we constructed synthetic data for three main categories: addition, removal, and replacement. (2) For drag editing, we utilized synthetic data to create scaling, translating, and rotating editing data. (3) For inpainting and outpainting, we randomly generate masks for each given image. (4) For reference image generation, We divided the tasks into subject-driven generation and image-conditioned generation ( i.e . , ControlNet-like generation) tasks. For subject-driven generation, we used a canvas filled with stickers, allowing the model to reference a specific sticker for generation, which helps train the model to extract the demanded objects or details from given images for generation. For the image-conditioned generation, we created canny, depth, and segment maps, similar to ControlNet. (5) We use synthetic data to enhance the model’s T2I accuracy in responding to attributes such as quantity, position, relationships, color, shape, and text. Concretely, we randomly arrange different quantities of stickers, text, and geometric shapes on a canvas, and obtain precise descriptions of their quantity, position, relationships, and colors based on their exact coordinates. During training, thanks to our unified framework that integrates T2I with multiple editing tasks, we can easily train T2I data alongside various types of editing data, obtaining a native unified image generation and editing model, called DreamOmni. The editing tasks benefit from the T2I data, which helps prevent the model from forgetting specific concepts and generation quality decline, while the T2I task can leverage synthetic data to enhance instruction following. • We conducted an analysis of existing model frameworks under fair settings and, drawing on the characteristics of different tasks, proposed an efficient and powerful unified image generation and editing framework, DreamOmni. • We introduced a synthetic collage data pipeline to tackle the current inefficiency and difficulty in creating and filtering high-quality editing data. Furthermore, we utilized the synthetic collage data pipeline to enhance the accuracy of the T2I model’s output. Experimental results demonstrate that synthetic data is an effective, high-quality, and cost-efficient method to scale up data for achieving unified image generation and editing training. • After unified training with T2I data and various synthetic datasets, our DreamOmni showcases competitive performance on T2I generation and various editing tasks. Figure 2 : The overview of DreamOmni. (a) The DreamOmni framework supports unified image generation and editing, with fast training convergence and powerful performance. (b) To overcome the difficulty and inefficiency in data creation and filtering for image editing, we propose a collage-based synthetic data pipeline. This pipeline enables the efficient creation of data for various editing tasks, such as adding, deleting, and replacement operations in instruction-based editing, as well as translation, scaling, and rotation in drag editing. Additionally, it supports reference image generation and segmentation \& detection. Furthermore, our synthetic data pipeline enhances the accuracy of T2I generation. Due to space limitations, we have optionally shown the corresponding prompts or instructions for these cases. 2 Related Work Text-to-image Diffusion Model. Diffusion models [ 52 ,

Read the complete text