Text-to-Image Models Need Less from Text Encoders Than You Think

Nurit Spingarn, Noa Cohen, Tamar Rott Shaham, Tomer Michaeli, [1pt] Technion -- Israel Institute of Technology · 2026

Read Text-to-Image Models Need Less from Text Encoders Than You Think by Nurit Spingarn, Noa Cohen, Tamar Rott Shaham, Tomer Michaeli, [1pt] Technion -- Israel Institute of Technology (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03715.txt.

Introduction

Spingarn and co-authors show that text-to-image models need far less from text encoders than assumed. Prompts are encoded into embeddings that guide image generation. Those embeddings carry rich context, like composition and attribute binding. Whether image models use that context was largely untested. The team asks which parts of the text signal are truly essential.

Two simple aspects turn out to suffice. Adjacent tokens must merge into word-level units. Word order must be marked, as positional embeddings do. A new embedding holds only word meanings plus order tags, with no context at all. It guides generation as well as full embeddings do. Visual quality and text fidelity match the standard setup. The image model itself decodes the complex language structure. Text encoder design can thus be simpler than current practice suggests.

Catalog record

CREATORNurit Spingarn, Noa Cohen, Tamar Rott Shaham, Tomer Michaeli, [1pt] Technion -- Israel Institute of Technology
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03715

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-to-Image Models Need Less from Text Encoders Than You Think Nurit Spingarn, Noa Cohen, Tamar Rott Shaham, Tomer Michaeli, [1pt] Technion -- Israel Institute of Technology Abstract Text-to-image models rely on text prompts as their primary interface to human intent. Prompts are encoded by a text encoder into embeddings that condition the image generation process. Beyond individual token meanings, text embeddings encode contextual information across the full prompt, such as compositionality and attribute binding. However, whether image models actually exploit this richer information remains underexplored. Here, we address the question: Which aspects of text representation are essential for image generation? We show that text-to-image diffusion transformer-based models commonly rely only on two relatively straightforward aspects of text representations: (i) the merging of adjacent tokens into a word representation, for words spanning multiple tokens, and (ii) word order, which is imprinted by the positional embedding of the text-encoder. To show this, we construct a new text embedding that encodes only individual word meanings and order but lacks any contextual information about the full prompt. We find that this bag of position-tagged words representation is sufficient to successfully guide image generation, achieving visual quality and text fidelity that are on par with full text embedding-guided generation. This demonstrates that, contrary to common belief, text-to-image models often do not use the rich information encoded in the text embedding beyond individual word meanings and word order. Instead, the decoding of complex linguistic structures is performed by the image model itself. 1 1 footnotetext: Equal contribution. Correspondence to [email protected] . Project Webpage: https://nsping13.github.io/contextless-TTI/ . 1 Introduction Text-to-image (TTI) models have seen remarkable progress in recent years, with systems now capable of producing highly realistic and semantically rich images from complex natural language prompts. Central to these systems is the text encoder, which serves as the interface between the user’s intent, which is described by a text prompt, and the text embedding that conditions the image model. As generation capabilities advanced, models adopted increasingly powerful text encoders. Early systems relied on CLIP-based conditioning [ 24 ] , while later models, such as Imagen [ 28 ] , introduced large pretrained language models, like T5 [ 25 ] , significantly improving generation quality on benchmarks such as DrawBench [ 28 ] . More recent systems extend this trend: models such as Stable Diffusion (SD) 3 [ 6 ] combine multiple text encoders, while architectures like FLUX.2 [ 15 ] incorporate large language models such as Qwen [ 36 ] as encoders. This progression reflects a widely held assumption: richer and more expressive text representations lead to improved image generation. Figure 1 : Contextless text embeddings are often enough. We find that when pretrained TTI models are conditioned on text embeddings that are stripped off of any contextual information, they maintain high visual quality and prompt adherence. This surprising behavior is exhibited even for complex prompts that involve attribute binding, spatial relations, and numeracy. We show that the capability of generating text-adherent images from a bag-of-words-like textual representation is enabled by the positional information encoded within each word embedding. This information allows the TTI model to indirectly decipher the word’s role within the prompt and thus to disambiguate between different sentences that comprise the same set of words, as illustrated in the figure. Our observations suggest that TTI models extract less information from the text embeddings than is commonly thought. Despite this progress, it remains unclear which aspects of these rich text representations are actually utilized by the image generation models. Modern text encoders are capable of capturing complex linguistic structure, including compositionality, attribute binding, and long-range dependencies across the input prompt [ 31 , 12 , 11 ] . However, the presence of such information in the text representation does not necessarily imply that it is functionally used by the image model. Here, we ask: which aspects of text representations are essential for guiding image generation? ( i ) BoT ( ii ) BoW ( iii ) BoPTW Figure 2 : Illustration of our three contextless text embeddings. We propose three embedding types of increasing richness: ( i ) Bag of Tokens, where each token is represented independently; ( ii ) Bag of Words, where tokens are merged into word-level representations; and ( iii ) Bag of Position-Tagged Words, where word embeddings additionally reflect their position in the prompt. To answer this question, we construct modified text embeddings that retain only specific aspects of the text prompt representation but contain no contextual information. We focus on three contextless text embedding types, as illustrated conceptually in fig. 2 : ( i ) Bag-of-Tokens (BoT), where the embedding contains information about each token separately, without any additional context from the full prompt, ( ii ) Bag-of-Words (BoW), where we represent words (that might be composed of multiple tokens) similarly without full-prompt context, and ( iii ) Bag-of-Position-Tagged-Words (BoPTW), where we let the word embeddings in the bag of words vary according to their order in the prompt. Importantly, our contextless embeddings serve as drop-in replacements for the original text encoder output, requiring no modifications to the pretrained image model. We construct them by replacing each token’s full embedding with an average over its embeddings in unrelated sentences, carefully controlling which sentences are included to isolate each specific aspect of the representation. We study how pretrained transformer-based diffusion models react to our contextless embeddings, focusing on SD 3 [ 6 ] , FLUX.1 Schnell [ 14 ] , and FLUX.2 Klein-4B [ 15 ] . We use prompts from the DrawBench [ 28 ] , GenEval [ 7 ] , and the MS-COCO 2014 validation set [ 19 ] , which represent diverse aspects of image generation, such as compositionality, counting, color fidelity, and attribute binding. The generated images are assessed by employing a VLM as a judge which is a well established methodology for evaluating image generation [ 38 , 5 , 22 , 16 , 17 ] . By comparing image generation conditioned on our simplified representations to generation with full text embeddings, we directly assess which aspects of text representations are required to guide image synthesis. We find that all contextless embedding variants achieve surprisingly strong results. For simple prompts, a large fraction of the cases require only BoT representations to achieve good results, suggesting that individual token meanings alone are often sufficient to guide generation. For complex prompts, the best results are achieved with the BoPTW embeddings. Despite lacking compositional structure, attribute binding, and other forms of contextual interaction between words, BoPTW achieves results that are on par with the full text embedding across a range of challenging scenarios. These include object counting, spatial relationships and attribute binding, as seen in Fig. 1 . These results demonstrate that, contrary to common belief, TTI models often do not use the rich contextual information encoded in text embeddings beyond individual word meanings and their order. This calls for a reframing of our understanding of the respective roles of the text encoder and image model, and raises questions about the prevailing drive toward increasingly large and complex text encoders. It also suggests that future text-to-image architectures may benefit from focusing on the image model’s capacity to interpret linguistic structure,

Read the complete text