Yuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blocks
Shaohua Wu, Tong Yu, Shenling Wang, Xudong Zhao · 2025
Read Yuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blocks by Shaohua Wu, Tong Yu, Shenling Wang, Xudong Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-16586.txt.
Introduction
Shaohua Wu and three co-authors upgrade text-to-image generation in a 2025 paper on arXiv. Diffusion models build images inside a U-shaped network whose convolutional blocks see only local neighborhoods, which can blind them to long-range meaning in a scene. The team swaps those blocks for Swin transformer blocks in both encoder and decoder.
Text conditioning gets equal care, through a well-chosen text encoder and a careful way of feeding text embeddings into the network. An adapted time step search across diffusion stages improved inference performance by a further 10 percent. The model reached an FID score of 1.37 on the ImageNet generation benchmark, state of the art at the time, with no auxiliary models at any denoising stage. In side-by-side tests, people struggled to tell its images from human-painted ones.
Catalog record
| CREATOR | Shaohua Wu, Tong Yu, Shenling Wang, Xudong Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-16586
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Yuan-TecSwin: A text conditioned Diffusion model with Swin-transformer blocks Shaohua Wu, Tong Yu, Shenling Wang, Xudong Zhao Abstract Diffusion models have shown remarkable capacity in image synthesis based on their U-shaped architecture and convolutional neural networks (CNN) as basic blocks. The locality of the convolution operation in CNN may limit the model’s ability to understand long-range semantic information. To address this issue, we propose Yuan-TecSwin, a text-conditioned diffusion model with Swin-transformer in this work. The Swin-transformer blocks take the place of CNN blocks in the encoder and decoder, to improve the non-local modeling ability in feature extraction and image restoration. The text-image alignment is improved with a well-chosen text encoder, effective utilization of text embedding, and careful design in the incorporation of text condition. Using an adapted time step to search in different diffusion stages, inference performance is further improved by 10%. Yuan-TecSwin achieves the state-of-the-art FID score of 1.37 on ImageNet generation benchmark, without any additional models at different denoising stages. In a side-by-side comparison, we find it difficult for human interviewees to tell the model-generated images from the human-painted ones. I Introduction Recently, text-guided synthesis has become a universal method for image generation and edition [ 30 ] [ 26 ] [ 27 ] . Based on large- scale multimodal datasets and scalability of the model, a series of generative models, including GAN, diffusion models, autoregressive model and flow model, have reached an unprecedented level on text- to-image synthesis and can even generate photorealistic images [ 30 ] [ 26 ] [ 11 ] [ 1 ] . Diffusion models benefit from the fixed learning process and the flexible model structure, and thus are widely applied in image synthesis. They are able to create images with high diversity without collapse, because of the ability to maintain semantic structure and fine-grained images. Compared to the diffusion models, Generative Adversarial Networks (GAN) [ 2 ] [ 16 ] usually suffer unstable training during adversarial training and difficulties in generating diversified images. Autoregressive models [ 38 ] and flow models often use an approximation as posterior probability and calculate in maximum likelihood. VAE have to use a surrogate loss, and the flow model relies more on a specific model structure. Fig. 1: Sample images generated by Yuan-TecSwin.(a) A painting in Chinese-style of a tiger couching in the bush (画一张国画,蹲在草丛中的老虎); (b) Make a painting in Impressionism. Describing the sunrise over the sea. (画一幅印象派油画,描绘了海上日出的场景); (c) A wedding cake with three layers, decorated with roses. (一个三层的婚礼蛋糕,蛋糕上装点着玫瑰花); (d) A classic Chinese painting about a basket of fruits, (一幅中国画,关于一篮子水果); (e) An oil painting of a women with blonde curly hair (画一幅油画,一个女人,长着金色的卷发); (f) This is an photo of living room with green wall. There are two couches in the living room with a tea table in front of them. There are plants on the tea table. (这是一张客厅的照片,客厅的墙是绿色的,摆着两个沙发,沙发前面有一个玻璃茶几,沙发上放着白色和绿色的靠枕,茶几上放着绿色植物); (g) Make an ink and wash painting. Describing the mountains over clouds. (画一幅水墨画,描绘了云雾中的山峦); (h)Corridor interior view of a greenhouse with many flowers planted and sunlight streaming into the greenhouse (一座温室的走廊内景,温室里种了很多花,有阳光照进温室中来); (i) An oil painting of a golden kitty with blue eyes (画一幅油画,一只金色的猫咪,猫咪的眼睛是蓝色的). Diffusion models with CNN-based blocks have achieved excellent result in image synthesis, with their capability in extracting low-level features such as the shape, color and texture of an object. However, in term of global or high-level visual information regarding to the relations and connections between objects, the CNN-based structure shows its weak points due to the capability in long-range semantic information. In a text-to-image generation task, it is important to regenerate basic features of an object, while the position, relative size, and depth of field are also significant elements to be considered. Motivated by the application of the transformer structure in NLP models, and the performance of Swin-transformer in computer vision domain, we propose a Swin-based Unet to improve the efficiency and balance the speed-performance trade-off in image generation. Compared to the words processed in NLP tasks, images, especially images with high resolution, has much higher resolution of pixels, which significantly increases the computational complexity. Swin-transformer combines the advantages of CNN and transformer, developed with a hierarchical feature map for a more computational-efficient adaption in the image domain. In addition to the model structure, the iterative sampling process from noise to meaningful images is also a key to the performance and inference time of the diffusion model. Nicole and Dhariwal [ 22 ] find that the diffusion model can maintain high sample quality with fewer steps, which speeds up sampling in practical applications. We also notice a common setting in recent diffusion models that most of which usually consider global time steps as equal when calculating the loss function [ 22 ] [ 15 ] . In fact, in the denoising time steps when the image is slightly corrupted, the task of image recovery can only contribute less to the model training. In contrast, in the denoising time steps where the image is severely corrupted, the model will be forced to solve the tougher task only relying on a few identifiable entities [ 7 ] . Open-source large-scale multimodel datasets lower the training threshold [ 32 ] [ 12 ] , but the presence of watermark, pornography, violence, and racial discrimination in datasets can also decrease the generation capability of the diffusion model and make it unsuitable for public release. To capture the characteristic of a large-scale dataset, the parameters of the main diffusion models reach billions to tens of billions, which increases training difficulty and computing resource consumption [ 11 ] [ 1 ] . In addition to data quality, the text encoder is another factor that influences the performance of text-to-image synthesis. It is common for a text-to-image model to employ a pre-trained language model to encode input texts into a sequence of embedding. In this work, we propose Yuan-TecSwin, a text-to-image swin-based diffusion model dedicated to an efficient learning of the denoising training process and a rational usage of dataset. The experiments show that the Swin-transformer based diffusion model achieves a performance similar to that of the CNN-based diffusion models, with only one-tenth of the parameters. Specifically, we build a Chinese multi-modal dataset with up to 360 million image-text pairs to ensure the diversity and the quality. The pre-trained base model is further finetuned with a human-labeled artistic dataset in higher quality to reinforce the ability in generating artworks. Instead of a uniform sampling during denoising time steps, the adapted time step searching is introduced to further strengthen the model for image recovery task. Figure 1 shows sample images generated by Yuan-TecSwin. To summarize, the main contributions of our work are as follows: 1. A novel text-conditioned diffusion model is constructed by introducing Swin-transformer blocks. The model we constructed can achieve a better image generation performance compared with a diffusion model with with similar architecture and CNN-blocks. 2. A versatile application of text encoder is discovered, that the average output of multiple layers brings a better result than the output of the final layer. 3. The sdapted time step searching method is proposed to refine the denoising process during diffusing steps, which allows a significant performance on image generation, without incorporating additional models. 4. A new state-of-the-art FID of 1.37 is achieved on ImageNet generation