The Cow of Rembrandt Analyzing Artistic Prompt Interpretation in Text-to-Image Models
2025
Read The Cow of Rembrandt Analyzing Artistic Prompt Interpretation in Text-to-Image Models (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23313.txt.
Introduction
Alfio Ferrara and co-authors ask how text-to-image models split a picture into content and style. Classic computer vision treats the two as separate, yet diffusion models get no such lesson during training. Using Rembrandt as a test case, the team studied a transformer-based image model as it painted from written prompts.
They read cross-attention heatmaps, which show which prompt words steer which pixels. Words naming things mostly shaped the regions holding those objects. Words naming a style mostly touched the background and surface texture. The split varied with the prompt and style requested, but its presence hints that the models pick up the content-style divide on their own. The team shares code, data, and a tool for viewing the maps.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23313
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Cow of Rembrandt Analyzing Artistic Prompt Interpretation in Text-to-Image Models Abstract Text-to-image diffusion models have demonstrated remarkable capabilities in generating artistic content by learning from billions of images, including popular artworks. However, the fundamental question of how these models internally represent concepts, such as content and style in paintings, remains unexplored. Traditional computer vision assumes content and style are orthogonal, but diffusion models receive no explicit guidance about this distinction during training. In this work, we investigate how transformer-based text-to-image diffusion models encode content and style concepts when generating artworks. We leverage cross-attention heatmaps to attribute pixels in generated images to specific prompt tokens, enabling us to isolate image regions influenced by content-describing versus style-describing tokens. Our findings reveal that diffusion models demonstrate varying degrees of content-style separation depending on the specific artistic prompt and style requested. In many cases, content tokens primarily influence object-related regions while style tokens affect background and texture areas, suggesting an emergent understanding of the content-style distinction. These insights contribute to our understanding of how large-scale generative models internally represent complex artistic concepts without explicit supervision. We share the code and dataset, together with an exploratory tool for visualizing attention maps at https://github.com/umilISLab/artistic-prompt-interpretation . Index Terms — text-to-image generation, diffusion models, cross-attention analysis, content-style disentanglement 1 Introduction Fig. 1 : (Top left) image generated by a txt2img model using the prompt “ A painting of a giraffe in the Analytical Cubism style. ” with the corresponding heatmaps for (top center) content and (top right) style components. (Bottom) image and heatmaps from “ A cow with Rembrandt style. ” Nowadays, we employ models that take a textual prompt as input and generate an image as output, an image which, in most cases, closely reflects the description provided in the prompt. These text-to-image ( txt2img ) models are trained on billions of images sourced from the internet, including artworks from open-access, labeled online repositories such as WikiArt [ 1 ] . Through this training, the models implicitly acquire knowledge of art, artists, and artistic movements [ 2 ] . This learned knowledge is often leveraged when the model is instructed, for instance, to generate “ a painting of a giraffe in the Analytical Cubism style. ” This leads to intriguing questions: How does the txt2img model interpret stylistic instructions? And how does it represent the interplay between the content to depict and the style to apply? Historically, the field of Computer Vision (CV) has treated the notions of content and style as orthogonal, assuming that content is independent of style, and vice versa [ 3 ] . However, such a distinction is not explicitly provided to diffusion models during training. Moreover, disentangling content from style goes beyond simply associating content with what is depicted and style with how it is depicted. This motivates a deeper investigation into how these models internally conceptualize and separate these elements. In this work, we focus on analyzing the behavior of transformer-based txt2img diffusion models when generating artworks, with a particular emphasis on how they encode the concepts of style and content . To achieve this, we leverage the models’ cross-attention heatmaps, which allow us to attribute each pixel in the generated image to specific tokens in the input prompt. This allows to isolate image regions primarily influenced by tokens describing content, and those influenced by tokens describing style. If these two regions overlap significantly, it may suggest that the model interprets content and style as similar or entangled; if they are distinct, it may indicate that the model conceptually separates the two. Figure 1 shows the cross-attention heatmaps corresponding to the words “ giraffe ” and “ Analytical Cubism ”, which define the content and style in the prompt “ A painting of a giraffe in the Analytical Cubism style ”. In this example, the two heatmaps highlight largely complementary regions: the content component “ giraffe ” activates areas corresponding to the animal itself, while the style component “ Analytical Cubism ” influences the rest of the image. This suggests a separation of influence between content and style components during image generation. However, this distinction is not evident in the bottom example of Figure 1 , which reveals an interesting behavior: the model appears to apply Rembrandt’s style by “dressing” the cow, since part of the activated region associated with the style component includes the cow’s clothing. The remainder of this paper is organized as follows: Section 2 reviews foundational concepts and related work in computer vision and content–style disentanglement. Section 3 outlines our methodology for investigating the model’s understanding of content and style. Section 4 describes our experimental setup and configurations. Section 5 presents our results and offers interpretations of how the model distinguishes (or conflates) content and style. Finally, Section 6 concludes the paper and discusses potential directions for future research. 2 Related Work In this section, we begin by introducing key concepts related to txt2img models and their underlying mechanisms. We then review relevant literature on the interpretability of these models. Finally, we introduce the task of content–style disentanglement and discuss prior work that addresses this challenge. txt2img generation and diffusion models. Our work centers on the study of txt2img diffusion models. These models learn to predict the amount of noise that has been progressively added to an image, transforming it into Gaussian noise during training. At inference time, the model reverses this process to generate an image from noise only [ 4 ] . In txt2img generation, diffusion models are conditioned on textual prompts using text encoders. A standard architecture for such models typically includes a language encoder (e.g., CLIP [ 5 ] ) for generating word embeddings, a variational autoencoder (VAE) [ 6 ] to translate between latent representations and images, and a time-conditional U-Net [ 7 ] for denoising latent vectors [ 8 ] . Interpretability of txt2img models. Given the increasing use and impact of these models, there is a growing need for interpretability tools tailored to txt2img diffusion architectures. Among the various approaches offered by the field of Explainable AI, our focus lies on cross-attention-based methods. These models rely on cross-attention mechanisms to connect the textual information embedded in input prompts with the visual content generated during denoising. This mechanism enables the generation of heatmaps that attribute regions of the image to specific words in the prompt. Several recent works leverage this property to explain diffusion-based models by producing cross-attention attribution maps [ 9 , 10 , 11 ] . These maps can be used to investigate how textual prompts shape visual outputs, analyze syntactic structures through head–dependent heatmap interactions, and explore semantic phenomena such as feature entanglement. Additionally, attention-based explanations have proven valuable for downstream tasks such as object segmentation, as they visually delineate what the model perceives as coherent objects [ 9 , 12 , 13 , 14 ] . Beyond segmentation, attention maps also facilitate the discovery of biases and inconsistencies in model behavior [ 11 ] . Content-style disentanglement. Distinguishing between style and content in art remains a longstanding challenge in the field of Computer Vision (CV) [ 3 ] . While content—the subject or objects depicted in an artwork—is typically straightforward to identify, style is far more