3DIS-flux: Simple and Efficient Multi-Instance Generation with DiT Rendering
Dewei Zhou, Ji Xie, Zongxin Yang, Yi Yang * · 2025
Read 3DIS-flux: Simple and Efficient Multi-Instance Generation with DiT Rendering by Dewei Zhou, Ji Xie, Zongxin Yang, Yi Yang * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05131.txt.
Introduction
Dewei Zhou and co-authors extend a decoupled image method to the FLUX model for multi-instance generation. Users want images with set layouts and traits per object. Adapter methods need retraining for each new base model.
The base method splits work into scene building and detail rendering. Depth maps guide the scene construction phase first. Pretrained depth control models then render the fine detail. Only scene building needs any adapter training at all. The new version swaps in a transformer-based rendering model. FLUX brings stronger rendering quality to the pipeline. Older versions used U-Net models for that stage. The swap shows the frame survives model upgrades.
Catalog record
| CREATOR | Dewei Zhou, Ji Xie, Zongxin Yang, Yi Yang * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05131
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
3DIS-flux: Simple and Efficient Multi-Instance Generation with DiT Rendering Dewei Zhou, Ji Xie, Zongxin Yang, Yi Yang * Abstract The growing demand for controllable outputs in text-to-image generation has driven significant advancements in multi-instance generation (MIG), enabling users to define both instance layouts and attributes. Currently, the state-of-the-art methods in MIG are primarily adapter-based. However, these methods necessitate retraining a new adapter each time a more advanced model is released, resulting in significant resource consumption. A methodology named Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which decouples MIG into two distinct phases: 1) depth-based scene construction and 2) detail rendering with widely pre-trained depth control models. The 3DIS method requires adapter training solely during the scene construction phase, while enabling various models to perform training-free detail rendering. Initially, 3DIS focused on rendering techniques utilizing U-Net architectures such as SD1.5, SD2, and SDXL, without exploring the potential of recent DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension of the 3DIS framework that integrates the FLUX model for enhanced rendering capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map controlled image generation and introduce a detail renderer that manipulates the Attention Mask in FLUX’s Joint Attention mechanism based on layout information. This approach allows for the precise rendering of fine-grained attributes of each instance. Our experimental results indicate that 3DIS-FLUX, leveraging the FLUX model, outperforms the original 3DIS method, which utilized SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in terms of both performance and image quality. Project Page: https://limuloo.github.io/3DIS/. Figure 1: Images generated using our 3DIS-FLUX. Based on the user-provided layout, 3DIS ( Zhou et al., 2024c ) generates a scene depth map that precisely positions each instance and renders their fine-grained attributes without the need for additional training, using a variety of foundational models. Specifically, 3DIS-FLUX employs the state-of-the-art FLUX model for rendering, which is capable of producing superior image quality and offering enhanced control. 1 Introduction With the rapid development of Diffusion Models ( Ho et al., 2020 ; Song et al., 2020 ; Zhou et al., 2023 ; Zhao et al., 2024b ; Lu et al., 2024a ; Lu et al., 2024b ; Xie et al., 2024 ) , contemporary models ( Rombach et al., 2022 ; Rombach et al., 2023 ; Podell et al., 2023 ; BlackForest, 2024 ) are capable of generating high-quality images. At the same time, there is a growing demand for more control over the generation process ( Zhang et al., 2023 ; Liang et al., 2024 ; Lu et al., 2023 ; Zhao et al., 2024a ) . One prominent area of research that has garnered increasing attention is Multi-Instance Generation (MIG) ( Zhou et al., 2024a ; Zhou et al., 2024b ; Wang et al., 2024 ; Li et al., 2023 ) , which seeks to ensure precise alignment of each instance’s position and attributes with user-defined specifications during the generation of multiple instances. Current strategies for Multi-Instance Generation (MIG) can be broadly classified into three categories: 1) Training-free methods , such as Multi-Diffusion ( Bar-Tal et al., 2023 ) and RAG-Diffusion ( Chen et al., 2024b ) , which employ multiple sampling steps for each instance and later merge them based on layout information to achieve spatial control. BoxDiff ( Xie et al., 2023 ) and TFLCG ( Chen et al., 2024a ) define a score function to guide the model’s sampling process, thereby enabling control over the layout. 2) Adapter-based methods , exemplified by MIGC ( Zhou et al., 2024a ) and InstanceDiffusion ( Wang et al., 2024 ) , which introduce layout information by training additional attention layers atop pre-trained models. 3) Text encoder fine-tuning methods , such as Reco ( Yang et al., 2023 ) and Ranni ( Feng et al., 2024 ) , which incorporate layout information directly into the input text and subsequently fine-tune the text encoder and the whole model to embed this spatial context into the generated output. Adapter-based methods are currently widely used due to their ability to provide strong control without the need to train the entire model. However, these approaches face two main challenges: 1) the need to retrain on different models. For example, methods like MIGC ( Zhou et al., 2024a ) , GLIGEN ( Li et al., 2023 ) , and InstanceDiffusion ( Wang et al., 2024 ) were initially trained on SD1.5, but as more advanced models such as SDXL and SD3 emerged, techniques like IFAdapter ( Wu et al., 2024 ) and CreatiLayout ( Zhang et al., 2024 ) had to be retrained accordingly. This process is resource-intensive and can be particularly difficult for users with limited GPU access. 2) Large-scale instance-level annotations are often hard to obtain. Each instance generation can be viewed as a Text-to-Image task, and high-quality instance-level data is generally more challenging to acquire than high-quality image-level data. To address the aforementioned challenges, the Depth-Driven Decouple Instance Synthesis (3DIS) ( Zhou et al., 2024c ) approach introduces a novel framework for Multi-Instance Generation. Instead of directly generating RGB images, 3DIS first trains a layout-to-depth model to produce a scene depth map. Then, 3DIS utilizes widely pre-trained depth control models, which only require image-level annotations, to generate images based on the layout provided by the generated scene depth map. Finally, 3DIS employs a training-free method to precisely render the attributes of each instance. The 3DIS approach offers two key advantages: 1) It requires the training of only a depth generation model, which can ignore many fine-grained attributes during training and does not necessitate high-quality visual fidelity. 2) The training-free rendering in 3DIS enables the use of various pre-trained models and better preserves the generative capabilities of large pre-trained models. The original 3DIS paper focused on the training-free rendering approach using models based on U-Net architectures, such as SD1.5 ( Rombach et al., 2022 ) , SD2 ( Rombach et al., 2023 ) , and SDXL ( Podell et al., 2023 ) . With the advancement of diffusion model techniques, the Diffusion Transformer (DiT) ( Li et al., 2024 ) architecture has demonstrated superior capabilities compared to traditional U-Net models. Specifically, FLUX has not only achieved significant improvements in image quality but has also enhanced control capabilities beyond those of previous models. As 3DIS is a flexible framework capable of quickly adapting to various new foundational models, we have extended it to propose 3DIS-FLUX, which leverages FLUX for training-free rendering, enabling stronger control and higher-quality image generation. Figure 2: The overview of 3DIS-FLUX. In line with 3DIS, the 3DIS-FLUX approach decouples image generation into two distinct stages: the creation of a scene depth map and the training-free rendering of high-quality RGB images using various generative models. 3DIS-FLUX utilizes the Layout-to-Depth model from 3DIS to generate the scene depth map, and subsequently employs the FLUX-depth model to render images based on the depth map. During this process, 3DIS-FLUX incorporates an Attention Controller to ensure the accurate fine-grained attributes of each instance. The overview of 3DIS-FLUX is depicted in Fig. 2 . Consistent with the original 3DIS framework, 3DIS-FLUX initially generates a scene depth map using a layout-to-image model. Subsequently, we utilize the FLUX.1-depth-dev model for depth-to-image conversion. Through