A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation
Andrew Z. Wang 1,3 and Songwei Ge 2 and Tero Karras 3 and Ming-Yu Liu 3 and Yogesh Balaji 3 and 1 University of Washington and 2 University of Maryland and 3 NVIDIA · 2025
Read A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation by Andrew Z. Wang 1,3 and Songwei Ge 2 and Tero Karras 3 and Ming-Yu Liu 3 and Yogesh Balaji 3 and 1 University of Washington and 2 University of Maryland and 3 NVIDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08210.txt.
Introduction
Andrew Wang and co-authors study whether modern decoder-only language models make better text encoders for image generation, in a 2025 paper. Many image models still use older encoders. The team build a standard training and testing pipeline to isolate the effect of the text input.
They train twenty-seven image models with twelve different encoders, varying how embeddings are pulled, which model is used, and its size. The common habit of taking last-layer embeddings gave weaker results. Averaging normalized embeddings across all layers did much better. The study gives builders clear guidance on how to draw text features from language models for image work.
Catalog record
| CREATOR | Andrew Z. Wang 1,3 and Songwei Ge 2 and Tero Karras 3 and Ming-Yu Liu 3 and Yogesh Balaji 3 and 1 University of Washington and 2 University of Maryland and 3 NVIDIA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08210
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation Andrew Z. Wang 1,3 and Songwei Ge 2 and Tero Karras 3 and Ming-Yu Liu 3 and Yogesh Balaji 3 and 1 University of Washington and 2 University of Maryland and 3 NVIDIA Abstract Both text-to-image generation and large language models (LLMs) have made significant advancements. However, many text-to-image models still employ the somewhat outdated T5 and CLIP as their text encoders. In this work, we investigate the effectiveness of using modern decoder-only LLMs as text encoders for text-to-image diffusion models. We build a standardized training and evaluation pipeline that allows us to isolate and evaluate the effect of different text embeddings. We train a total of 27 text-to-image models with 12 different text encoders to analyze the critical aspects of LLMs that could impact text-to-image generation, including the approaches to extract embeddings, different LLMs variants, and model sizes. Our experiments reveal that the de facto way of using last-layer embeddings as conditioning leads to inferior performance. Instead, we explore embeddings from various layers and find that using layer-normalized averaging across all layers significantly improves alignment with complex prompts. Most LLMs with this conditioning outperform the baseline T5 model, showing enhanced performance in advanced visio-linguistic reasoning skills. 1 Introduction The field of text-to-image generation has seen tremendous progress in recent years, driven by improvements in diffusion architectures Esser et al. 2024 ; Chen et al. 2023 ; Liu et al. 2024a , scalable models Podell et al. 2023 ; Peebles and Xie 2023 , and better training procedures Liu et al. 2024a ; Karras et al. 2022 ; Karras et al. 2024 . Central to the success of these models is the use of pre-trained text encoders that transfer natural language prompts into representations suitable for guiding image generation. However, the impact of different text encoder models on the image generation quality remains largely unexplored. Figure 1 : VQA scores for text-to-image models using T5-XXL and Mistral-7B as the text encoders. The models use embeddings extracted from the last layer of T5 ( orange ) and Mistral ( green ), and layer-normalized average embeddings of Mistral ( blue ). Our results show that Mistral performs worse than T5 when only using the last layer. However, using all layers significantly improves Mistral’s performance, surpassing T5 in all aspects. Most contemporary image generators use traditional text encoders such as T5 Raffel et al. 2020 or CLIP Radford et al. 2021 for obtaining text embeddings Podell et al. 2023 ; Saharia et al. 2022 ; Balaji et al. 2022 ; Ramesh et al. 2022 ; Atzmon et al. 2024 . However, the development of encoder-decoder models has slowed down while there is an increasing interest in decoder-only large language models (LLMs) due to their great scalability. This raises a pressing question of whether these decoder-only LLMs are suitable for text-to-image generation tasks. In this work, we examine how the rich linguistic representations from decoder-only LLMs can be leveraged to enhance text-to-image generation and investigate the most effective strategies for incorporating them. Our focus is on autoregressive language models that use causal attention and are trained with a next-token prediction objective. We address three primary research questions: 1. To what extent can decoder-only LLMs be utilized to enhance text-to-image generation? What methodologies yield the largest improvements? 2. LLM fine-tuned embedding models have surpassed encoder-decoder models in contextual semantic comprehension across various tasks Muennighoff et al. 2022 . Can these embedding models contribute to improving text-to-image alignment? 3. Does increasing the model size of decoder-only LLMs lead to measurable improvements in the text-to-image generation performance? To address these questions, we develop a standardized text-to-image training and evaluation pipeline based on the Stable Diffusion v2 (SD2) Rombach et al. 2022 architecture. In each experiment, only the text encoder is replaced, while the rest of the architecture and training recipe remains unchanged. This allows us to isolate and understand the effect of different text representations. We explore a wide variety of models as text encoders, including two traditional text-to-image text encoders, seven open-source LLMs, and three fine-tuned embedding models. We quantitatively evaluate the models using VQAScore Li et al. 2024 on GenAI-Bench Li et al. 2024 , and conduct in-depth analysis to understand the strengths and limitations of using LLMs as the text encoders. Our main findings are as follows: • Using text embeddings from the last layer of the LLM is sub-optimal : To our knowledge, all current text-to-image diffusion models utilize the embeddings from the final layer of the text encoders as the conditional embedding Saharia et al. 2022 ; Balaji et al. 2022 . However, our results reveal that this approach does not translate effectively to LLMs, leading to inferior results compared to using T5. • Aggregating features from multiple layers outperforms using a single layer : We find that using embeddings normalized and averaged across all layers yields far better performance than relying on any single layer alone (Figure 1 ). This is because each layer within an LLM captures different aspects of linguistic information, so using averaged embeddings can combine the strengths of every layer to create a richer and more comprehensive representation Jawahar et al. 2019 ; Dar et al. 2022 ; Liu et al. 2024b . • LLM-based embedding models sometimes outperform the base models : Our results suggest that fine-tuned embedding models hold the potential for improving text-to-image generation. However, such a performance improvement is not always observed across all the embedding and base models. Despite these mixed outcomes, strong results are achieved with bge-Gemma2 Chen et al. 2024b using layer-normalized average embeddings, our best-performing model, highlighting promise in leveraging embedding models for text-to-image generation. • Scaling up the LLM is beneficial, but not across all aspects : Increasing the model size of LLMs consistently leads to improved performance. However, we observe that model size does not uniformly enhance all aspects of compositional text-to-image generation. These results suggest that simply scaling model size may not be the most efficient approach for improving performance across all skills, highlighting the potential of alternative strategies, such as hybrid models or skill-specific fine-tuning. 2 Related work CLIP ( last layer ) T5-XXL ( last layer ) Mistral ( norm avg ) bge-Gemma2 ( norm avg ) A single, vibrant red rose blooms defiantly through a narrow crack in the weathered, grey concrete . Its velvety petals unfurl gracefully, reaching for the sunlight that filters weakly through the urban haze. A small dog not in a tiny sweater, playing joyfully without any clothes . The fluffy white dog, with big brown eyes and floppy ears, bounds through a sun-drenched field of wildflowers. Its tongue lolls out in pure happiness as it chases a bright red butterfly, its tiny paws barely touching the ground. Figure 2 : Visual comparison of images generated with different text encoders. We use last-layer embeddings ( last layer ) from the text encoders of CLIP-ViT-H/14 (354M) and T5-XXL (4.7B). We also use average layer-normalized embeddings ( norm avg ) from the pre-trained LLM Mistral-7B (7B) and the fine-tuned embedding model bge-Gemma2 ( bge-multilingual-gemma2 ; 9B). Mistral and bge-Gemma2 can handle complex reasoning tasks such as negation (panel below) compared to models