U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers
Zhanjie Zhang, Ao Ma, Ke Cao, Jing Wang, Shanyuan Liu, Yuhang Ma, Bo Cheng, Dawei Leng, Yuhui Yin, * Equal Contribution, Corresponding Authors · 2025
Read U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers by Zhanjie Zhang, Ao Ma, Ke Cao, Jing Wang, Shanyuan Liu, Yuhang Ma, Bo Cheng, Dawei Leng, Yuhui Yin, * Equal Contribution, Corresponding Authors (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08157.txt.
Introduction
Zhanjie Zhang and eight colleagues present U-StyDiT, an artistic style transfer method built on diffusion transformers. The goal is repainting very high-quality content images in a learned artistic style. Earlier reconstruction-based and disentanglement-based methods leave artifacts and disharmonious patterns at that quality level.
A multi-view style modulator reads the style image from local and global perspectives to condition generation. A dedicated block learns content and style conditions simultaneously for cleaner disentanglement. The team also built Aes4M, a dataset of four million style images across ten categories, attacking the data bottleneck directly. Experiments show higher-quality stylized images than state-of-the-art methods, which the authors call a first for transformer-based diffusion.
Catalog record
| CREATOR | Zhanjie Zhang, Ao Ma, Ke Cao, Jing Wang, Shanyuan Liu, Yuhang Ma, Bo Cheng, Dawei Leng, Yuhui Yin, * Equal Contribution, Corresponding Authors |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
U-StyDiT: Ultra-high Quality Artistic Style Transfer Using Diffusion Transformers Zhanjie Zhang, Ao Ma, Ke Cao, Jing Wang, Shanyuan Liu, Yuhang Ma, Bo Cheng, Dawei Leng, Yuhui Yin, * Equal Contribution, Corresponding Authors Abstract Ultra-high quality artistic style transfer refers to repainting an ultra-high quality content image using the style information learned from the style image. Existing artistic style transfer methods can be categorized into style reconstruction-based and content-style disentanglement-based style transfer approaches. Although these methods can generate some artistic stylized images, they still exhibit obvious artifacts and disharmonious patterns, which hinder their ability to produce ultra-high quality artistic stylized images. To address these issues, we propose a novel artistic image style transfer method, U-StyDiT, which is built on transformer-based diffusion (DiT) and learns content-style disentanglement, generating ultra-high quality artistic stylized images. Specifically, we first design a Multi-view Style Modulator (MSM) to learn style information from a style image from local and global perspectives, conditioning U-StyDiT to generate stylized images with the learned style information. Then, we introduce a StyDiT Block to learn content and style conditions simultaneously from a style image. Additionally, we propose an ultra-high quality artistic image dataset, Aes4M, comprising 10 categories, each containing 400,000 style images. This dataset effectively solves the problem that the existing style transfer methods cannot produce high-quality artistic stylized images due to the size of the dataset and the quality of the images in the dataset. Finally, the extensive qualitative and quantitative experiments validate that our U-StyDiT can create higher quality stylized images compared to state-of-the-art artistic style transfer methods. To our knowledge, our proposed method is the first to address the generation of ultra-high quality stylized images using transformer-based diffusion. Figure 1 : Artistic style transfer results by the proposed U-StyDiT. Given a content and style image, our proposed U-StyDiT successfully produces ultra-high quality style transfer results that preserve the structure of the content image and the style information of the style image. 1 Introduction Given a content image, ultra-high quality artistic style transfer refers to transferring the learned style information onto this content image, generating an ultra-high quality artistic stylized image. Existing artistic style transfer methods can be primarily categorized into two types: style reconstruction-based [ 47 , 33 , 41 , 35 , 15 , 44 , 49 , 36 , 6 ] and content-style disentanglement-based style transfer approaches [ 46 , 38 , 43 , 39 , 13 ] . More specifically, style reconstruction-based style transfer approaches usually train a style adapter only on style images to condition stable diffusion (SD) [ 29 , 11 , 22 , 23 ] to generate desired images with the learned style information. For example, Ip-adapter [ 41 ] introduced a style adapter on Unet-based stable diffusion to learn style information from a style image. Recently, FLUX [ 20 ] , trained using DiT-based stable diffusion [ 26 ] , has shown outstanding capability in producing high-quality images. InstantX trained a style adapter on FLUX to inject style information. However, during the implementation of style transfer, these methods use a Canny image [ 5 ] as an additional condition to preserve the content structure of the stylized images. We argue that since the style condition and Canny are typically trained on separate style and content image datasets, this inevitably leads to information confusion when used simultaneously, introducing obvious artifacts and disharmonious patterns (e.g., in the 4^{th} col of Fig. 2 ). The content-style disentanglement-based style transfer approaches simultaneously focus on training content and style conditions. For example, CSGO [ 39 ] created a triad dataset that includes content images, style images, and stylized images. Training the model to decouple the stylized images into the corresponding content and style conditions can facilitate the generation of stylized images using both content and style images in inference. StyleShot [ 13 ] trained a Mixture-of-Expert (MoE) [ 28 ] module on a pre-trained content condition to extract multi-level style embeddings. However, although these methods learn the disentanglement of content and style to some degree, they still cannot produce ultra-high quality stylized images. We believe that the main reasons for this limitation are twofold: 1) Compared to transformer-based diffusion, the Unet-based diffusion has certain limitations in capturing long-range dependencies among patches, which hinders its ability to generate ultra-high quality stylized images. 2) Lack of ultra-high quality artistic style images and effective methods for model to learn content-style disentanglement. Figure 2 : We present some stylized image examples generated by our proposed U-StyDiT, InstantX [ 33 ] and CSGO [ 39 ] . Existing image stylization methods fail to create ultra-high quality stylized images, introducing obvious artifacts and disharmonious patterns. Figure 3 : Compared to Wikiart [ 24 ] , Aes4M has clearer Canny images. To solve the above problems, we introduce a novel framework, U-StyDiT, to generate ultra-high quality stylized images. First, we focus on designing a modulator to inject style information into a transformer-based diffusion model [ 21 ] . In detail, we propose a novel Multi-view Style Modulator (MSM) to extract style information from a style image from both local and global perspectives. Specifically, we simultaneously extract style information from the entire style image and multiple patches within the style image. Unlike previous methods [ 42 , 48 ] , we do not aggressively discard style image patches that have lower similarity to the overall image; instead, we merge these patches at the token level and sample style information to reduce computational complexity. Furthermore, we introduce a StyDiT Block to replace the existing DiT block, addressing the challenge of learning content and style conditions simultaneously based on transformer-based stable diffusion. To our knowledge, this is the first paper that simultaneously trains content and style conditions on transformer-based stable diffusion to accomplish the task of generating ultra-high quality stylized images (e.g., We present some ultra-high quality stylized image samples generated by our method in Fig. 1 ). In addition, we find that existing methods typically use Canny [ 5 ] to extract content conditions. However, Canny often encounters difficulties when applied to existing artistic images, such as those from Wikiart [ 24 ] . This is because artistic images in Wikiart tend to have a high frequency of detailed texture, making it challenging to extract Canny images and style information from human-created artistic collections to train content and style conditions. To this end, we propose an ultra-high quality artistic image dataset, Aes4M, which consists of 10 categories, with 400,000 artistic images in each category, totaling 4 million artistic images. Each artistic style image has high aesthetics and text-image consistency, as well as clear Canny image. In conclusion, we show our contributions as follows: • We propose a novel artistic image style transfer method, U-StyDiT, which is built on transformer-based diffusion (DiT) and can generate ultra-high quality artistic stylized images. • We propose a novel Multi-view Style Modulator (MSM) to extract style information from a style image from both local and global perspectives and introduce a StyDiT Block to simultaneously address the challenge of learning content and style conditions on transformer-based stable diffusion. • We build an ultra-high quality artistic image dataset, Aes4M, consisting of 10 categories, with 400,000 artistic images in each category, totaling 4 million. Unlike