UniGP : Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception

Qin Guo, Hao Luo, Dongxu Yue, Weixuan Jin, Xiao Fu, Fan Wang, Dan Xu, [4pt] The Hong Kong University of Science, Technology · 2026

Read UniGP : Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception by Qin Guo, Hao Luo, Dongxu Yue, Weixuan Jin, Xiao Fu, Fan Wang, Dan Xu, [4pt] The Hong Kong University of Science, Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30332.txt.

Introduction

Qin Guo, Hao Luo, Dongxu Yue, Weixuan Jin, Xiao Fu, Fan Wang, and Dan Xu unify image generation and dense prediction inside one diffusion transformer. Controllable generation makes pictures obeying depth or normal maps. Dense prediction estimates those maps from pictures. The two tasks are usually trained as separate models. Their framework, UniGP, trains both jointly on shared foundations.

UniGP builds on the MMDiT architecture without task-specific designs or losses. A copied image branch models dense outputs beyond plain color images. A unified dataset strategy mixes heterogeneous training data together. The model learns the joint distribution of image and geometry pairs. It can generate, predict, and do both jointly. Experiments show it surpassing earlier unified approaches overall. It performs on par with specialized single-task methods too. Joint training helps both directions: generation gains structural alignment. Perception gains detail from the generative priors.

Catalog record

CREATORQin Guo, Hao Luo, Dongxu Yue, Weixuan Jin, Xiao Fu, Fan Wang, Dan Xu, [4pt] The Hong Kong University of Science, Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.30332

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UniGP : Taming Diffusion Transformer for Prior-Preserved Unified Generation and Perception Qin Guo, Hao Luo, Dongxu Yue, Weixuan Jin, Xiao Fu, Fan Wang, Dan Xu, [4pt] The Hong Kong University of Science, Technology Abstract Recent advances in diffusion models have shown impressive performance in controllable image generation and dense prediction tasks. However, existing approaches typically treat diffusion-based controllable generation and dense prediction as separate tasks, overlooking the potential benefits of jointly modeling the heterogeneous distributions. In this work, we introduce UniGP , a framework built upon MMDiT, which unifies controllable generation and dense prediction through simple joint training, without the need for complex task-specific designs or losses, while preserving the backbone’s versatile priors. By learning controllable generation and prediction under different conditions, our model effectively captures the joint distribution of image-geometry pairs. UniGP is capable of versatile controllable generation, dense prediction, and joint generation. Specifically, the proposed UniGP consists of DUGP and a unified dataset training strategy. The former, following the principle of Occam’s razor, uses only a copied image branch of MMDiT to model dense distributions beyond RGB, while the latter integrates heterogeneous datasets into a unified training framework to jointly model generation and perception tasks. Extensive experiments demonstrate that our unified model surpasses prior unified approaches and performs on par with specialized methods. Furthermore, we demonstrate that multi-task joint training provides complementary benefits: generative priors enrich perceptual details, while perceptual learning improves structural alignment in generation. Project page: guoqincode.github.io/UniGP . Figure 1: We present UniGP , a Diffusion Transformer-based framework that simultaneously models RGB and dense distributions within a single framework, supporting: (a) Text to image, depth, and surface-normal joint generation; (b) Joint depth and surface-normal estimation; and (c) Any-condition text-to-image generation. 1 Introduction Large-scale Text-to-Image (T2I) Diffusion models rombach2022high ; sd3 ; Midjourney have recently demonstrated exceptional high-quality image generation. This success has spurred exploration into downstream tasks based on pretrained T2I models, falling into two major categories: 1) Controllable generation, where models like ControlNet zhang2023adding use external conditions ( e.g. depth, normal) to guide the T2I process. 2) Dense prediction, where models are repurposed for tasks like monocular depth estimation ke2023repurposing , later extended to normal estimation ye2024stablenormal and joint depth-normal estimation garcia2024fine ; fu2025geowizard . Although these methods have achieved remarkable results independently, they only model the transition within a single distribution, limiting them to simple, single-task scenarios. One might ask: since regression-based feed-forward models already achieve highly accurate dense predictions, why should we pursue perception within a generative framework? The answer lies in the fundamental limitation of regression models: they act as passive, unidirectional observers. While they predict geometry accurately, they cannot internalize this geometric consciousness to actively guide complex creation processes. A truly unified model should embed perception into the generative latent space, allowing geometric understanding to act as a core generative prior. Previous work, JointNet zhang2024jointnet , aimed to model multiple distributions simultaneously through intricate architectural design. While this approach resulted in a significant increase in parameter count, it achieved suboptimal results. Recent works including UniCon li2024unicon , OneDiffusion le2025one , and JoDi xu2025jodi explored how to model the joint distribution but largely ignored the complementary mechanism between perception and generation tasks, resulting in performance gaps compared to diffusion-based expert models. In this paper, we propose UniGP , a unified diffusion transformer (DiT) model that learns the joint distribution of different modalities of an image with a flexible architecture. First, we adopt the Multi-Modal DiT (MMDiT) framework sd3 . Following the principle of Occam’s razor, we introduce the d isentangled u nified g eneration and p erception branch ( DUGP ), which only copies the image branch from MMDiT to model distributions beyond RGB, gracefully avoiding the massive computational overhead of replicating the entire backbone. Second, we propose a unified dataset and training strategy to bridge these heterogeneous tasks. By employing a simple binary loss weighting, our model jointly learns generation and perception, obviating the need for task-specific designs. Third, we demonstrate the complementary benefits between tasks. Our ablation studies show that jointly training perception and generation within a single framework allows perceptual learning to enforce strict spatial alignment for generation, while generative priors imbue perception with finer details. Compared to representative controllable generation methods, UniGP achieves superior controllability with less data. Unlike mainstream dense prediction methods that quickly suffer from catastrophic forgetting, our approach preserves generative capacity, which enhances perceptual generalization. As shown in Fig. 1 , UniGP supports various tasks within a single model: 1) text-to-image, depth and surface-normal joint generation; 2) joint depth and surface-normal estimation; and 3) any-condition text-to-image generation. To summarize, our main contributions are three-fold: • We propose UniGP , an MMDiT framework that unifies generation and perception via a DUGP branch. Following Occam’s razor, this branch models non-RGB data by reusing only the backbone’s image branch parameters. • We propose a unified dataset and a simple training strategy with a binary loss, enabling joint learning of heterogeneous tasks without task-specific architectures. • Extensive experiments show UniGP significantly surpasses prior joint models and rivals task-specific experts, demonstrating clear complementary benefits between jointly trained perception and generation tasks. 2 Related Work Controllable Diffusion Models. Controllable diffusion models aim to use external conditions to guide generation. ControlNet zhang2023adding and its subsequent models qin2023unicontrol ; zhao2024uni ; sun2024anycontrol extend T2I generation by encoding condition signals into latent representations using a trainable UNet encoder, injected into the backbone via zero convolution. However, these methods are limited strictly to generation tasks and require massive datasets for precise control. In contrast, our method jointly trains controllable generation and dense prediction tasks, enabling faster convergence and superior geometric consistency with significantly less data. We propose a novel controlling module design and training strategy for MMDiT, enhancing capabilities within the current SOTA diffusion framework. Figure 2: Framework of UniGP . 1) Our inputs include: a) RGB images; b) Depth and Normal images; and c) Randomly selected condition as described in Section 3.1 . d) Prompts (omitted for brevity). 2) After VAE encoding and adding noise, the noisy RGB, depth and normal latents are fed into the backbone’s patcher, while the clean condition latents are passed to the Condition Patcher of DUGP . Then, the tokens of noisy depth/normal and condition are concatenated token-wise and pass through the Stacked Control Layers and Stacked Perception Layers . 3) Finally, the backbone generates RGB images, while DUGP generates depth and normal maps. Diffusion Models in Perception Tasks. Currently, a notable trend involves adapting pretrained T2I diffusion models for dense prediction tasks vandenhende2021multi , such as depth and surface normal estimation. Marigold ke2023repurposing fine-tuned Stable Diffusion (SD) rombach2022high to generate precise depth maps, leveraging SD’s strong geometric priors. Follow-up works gui2024depthfm ; garcia2024fine ; he2024lotus ; ye2024diffusionmtl ; fu2025geowizard ; xu2024diffusion improve its performance and efficiency. StableNormal ye2024stablenormal extends this paradigm to surface normal estimation. GeoWizard fu2025geowizard jointly predicts depth and normal using parallel branches. The above models entirely fine-tune diffusion for perception tasks, quickly losing versatile generative priors and narrowing their applicability. Unlike prior works that compromise generation when adapted to perception, our method explicitly bridges generative and perception distributions, quarantining generative priors while learning highly accurate perception. Unified Diffusion Models. While less common, some works have

Read the complete text