Trung X. Pham

Zhang Kang · 2025

Read Trung X. Pham by Zhang Kang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09164.txt.

Introduction

Customizing product photos with a chosen object, without fresh training for each object, is the task tackled by Pham and co-authors in a 2025 paper. Older tools for this job use heavy networks that cost much memory and time. The new frame uses a light masked diffusion model that works on compact image codes instead.

Three parts do the work: the light model itself, a split condition design that keeps backgrounds aligned, and a collector that packs many inputs into a small set. Tests on a fashion dataset beat the prior best method on all image quality scores used. The frame also uses fewer parameters and less memory, the authors report.

Catalog record

CREATORZhang Kang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09164

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trung X. Pham Zhang Kang Abstract We propose E-MD3C ( E fficient M asked D iffusion Transformer with Disentangled C onditions and C ompact C ollector), a highly efficient framework for zero-shot object image customization. Unlike prior works reliant on resource-intensive Unet architectures, our approach employs lightweight masked diffusion transformers operating on latent patches, offering significantly improved computational efficiency. The framework integrates three core components: (1) an efficient masked diffusion transformer for processing autoencoder latents, (2) a disentangled condition design that ensures compactness while preserving background alignment and fine details, and (3) a learnable Conditions Collector that consolidates multiple inputs into a compact representation for efficient denoising and learning. E-MD3C outperforms the existing approach on the VITON-HD dataset across metrics such as PSNR, FID, SSIM, and LPIPS, demonstrating clear advantages in parameters, memory efficiency, and inference speed. With only 1 4 \frac{1}{4} of the parameters, our Transformer-based 468M model delivers 2.5 × 2.5\times faster inference and uses 2 3 \frac{2}{3} of the GPU memory compared to an 1720M Unet-based latent diffusion model. marginparsep has been altered. topmargin has been altered. marginparpush has been altered. The page layout violates the ICML style. Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you. We’re not able to reliably undo arbitrary changes to the style. Please remove the offending package(s), or layout-changing commands and try again. E-MD3C: Taming Masked Diffusion Transformers for Efficient Zero-Shot Object Customization † † footnotetext: * Equal contribution . Correspondence to: Trung X. Pham <[email protected]>, Chang D. Yoo <[email protected]>. KAIST, South Korea , Copyright 2025 by the author(s). 1 Introduction We address the challenge of zero-shot object-level image customization (ZSOIC) Chen et al. (2024b) , a task that has gained traction with the rise of diffusion models and open-source frameworks like Stable Diffusion Rombach et al. (2022) ; Zhang et al. (2023) . Existing methods, including ObjectStitch Song et al. (2023) and Paint-by-Example Yang et al. (2023) , focus on localized edits but struggle with identity consistency, especially for unseen categories. Customization techniques such as Textual Inversion Gal et al. (2022) , DreamBooth Ruiz et al. (2023) , and UniCanvas Jin et al. (2025) enable novel concept generation but lack spatial control and require extensive fine-tuning, limiting their real-world applicability. AnyDoor Chen et al. (2024b) mitigates these issues by introducing a Stable Diffusion-based pipeline that ensures ID consistency in zero-shot settings, achieving flexible, high-quality region-specific edits using ID tokens and frequency-aware detail maps. However, its reliance on resource-heavy Unet-based architectures with ControlNet Zhang et al. (2023) significantly increases computational overhead (see Fig. 1 ). Notably, prior works have largely overlooked the efficiency and potential of Vision Transformers (ViTs) Dosovitskiy et al. (2021) , which offer a compelling alternative to Unet-based designs. This raises a key question: Can we achieve competitive performance while substantially reducing computational costs? Figure 1: Existing Approach Inefficiency. The current model (black) demands significant parameters, memory, and inference time due to its resource-intensive architecture. Inspired by recent advances in transformer-based diffusion models, such as DiT Peebles & Xie (2023) ; Esser et al. (2024) ; Xie et al. (2025) and masked diffusion transformers Gao et al. (2023) ; Pham et al. (2024) ; Mao et al. (2024) ; Pham et al. (2025) , we explore their efficiency and adaptability for ZSOIC. Diffusion transformers Peebles & Xie (2023) offer stable training and strong generative performance, outperforming traditional Unet-based approaches Ho et al. (2020) ; Rombach et al. (2022) , making them well suited for this task. We introduce E-MD3C , a novel framework that employs masked diffusion transformers Gao et al. (2023) for ZSOIC. Our approach features a specialized disentangled collection network that consolidates conditions into a compact vector for adaptive LayerNorm modulation, alongside a masking network to enhance transformer learning. Despite its simple yet effective design, E-MD3C surpasses existing methods in FID, SSIM, L1, and LPIPS on the VITON-HD dataset Choi et al. (2021) , while significantly reducing parameter count and computational demands. Our main contributions are as follows: • We propose E-MD3C , the first masked diffusion transformer-based model for zero-shot object customization, achieving high efficiency through latent patches and compact multi-condition representations. • A novel Conditions Collector Module (CCM) is introduced to consolidate essential information into a lightweight representation for denoising diffusion, significantly reducing computational complexity compared to methods that concatenate full conditions with noisy target images. • Our framework decouples conditions into two branches: the hint image supervises the target image in the denoising branch, while other conditions are processed via the CCM and a single cross-attention layer. This design enhances token alignment, preserves background details, and ensures robust correspondence between target and source images. • E-MD3C outperforms the heavy Unet-based approach across multiple metrics, using only a quarter of the parameters and achieving 2.5\times faster inference. 2 Related Works Zero-shot Object Customization. Chen et al. (2024b) introduced AnyDoor to tackle the ZSOIC problem, utilizing a large pre-trained diffusion model, Stable Diffusion Rombach et al. (2022) , with a dual-backbone architecture inspired by ControlNet Zhang et al. (2023) . Despite achieving strong performance, AnyDoor relies on a resource-intensive architecture with significant computational demands, making the image generation process inefficient. Similarly, Chen et al. (2024a) proposed MimicBrush, a Unet-based latent diffusion model that integrates a reference U-Net into an imitative U-Net, offering finer part-level control but requiring a large model size and heavy computational resources. UniCanvas Jin et al. (2025) also employs a substantial framework built upon customized text-to-image generation with Stable Diffusion, introducing a novel affordance-aware editing pipeline but further amplifying computational overhead with its parallel generative branches. While these approaches improve object customization applications, their reliance on oversized CNN-based architectures limits practical usage and applicability, especially on hardware with limited resources. To address these shortcomings, we propose E-MD3C , a lightweight transformer-based diffusion model that generates high-quality images while significantly reducing resource requirements (see Fig. 1 ). Figure 2: Object Composition . The 3 {}^{\text{rd}} and 4 {}^{\text{th}} columns show outputs from the existing method and our model. Our model generates images in just over 2 seconds, compared to 7 seconds for the existing approach. Denoising Diffusion Transformers. The CNN-based U-Net architecture Ronneberger et al. (2015) has long been the foundational framework for diffusion models and remains a prevalent choice for various diffusion-based generation tasks Ho et al. (2020) ; Ho et al. (2022) ; Song & Ermon (2019) . However, a transformative shift occurred with the introduction of DiT Peebles & Xie (2023) , which incorporated the transformer-based ViT architecture Dosovitskiy et al. (2021) into latent diffusion models. This innovation demonstrated superior scalability and consistently outperformed traditional U-Net-based designs. Building on this progress, Gao et al. (2023) further advanced diffusion transformers, achieving state-of-the-art results in class-conditional image generation on ImageNet through advanced contextual representation learning. Other works, such as Stable Diffusion v3 Esser et al. (2024) , PixArt Chen et al. (2025) and Sana Xie et al. (2025) , have leveraged DiT for text-to-image generation, highlighting the strong potential of transformer architectures in generative tasks. Although prior studies primarily focus on general-purpose or text-driven generative tasks, our research investigates the application of diffusion transformers to the specialized and complex task of

Read the complete text