Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor
Vatsal Agarwal, Matthew Gwilliam, Gefen Kohavi, Eshan Verma, Daniel Ulbricht, Abhinav Shrivastava · 2025
Read Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor by Vatsal Agarwal, Matthew Gwilliam, Gefen Kohavi, Eshan Verma, Daniel Ulbricht, Abhinav Shrivastava (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07106.txt.
Introduction
Vatsal Agarwal and co-authors test, in a 2025 paper, if a text-to-image model can serve as the eyes of a talk model. The usual eyes, CLIP, catch the broad scene but miss fine points that a query needs. Spread models, the team finds, hold rich sense marks and tight image-text ties in their guts. Text cues can aim those marks at the parts a query asks for. They then join the marks to a talk model and hit a snag: the talk model can leak the source prompt back. Causes are read, and a fix is set.
A blend of CLIP marks and cued spread marks is their end pick. Gains show most on sight-led tasks of space and build.
Catalog record
| CREATOR | Vatsal Agarwal, Matthew Gwilliam, Gefen Kohavi, Eshan Verma, Daniel Ulbricht, Abhinav Shrivastava |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07106
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Multimodal Understanding via Stable Diffusion as a Task-Aware Feature Extractor Vatsal Agarwal, Matthew Gwilliam, Gefen Kohavi, Eshan Verma, Daniel Ulbricht, Abhinav Shrivastava Abstract Recent advances in multimodal large language models (MLLMs) have enabled image-based question-answering capabilities. However, a key limitation is the use of CLIP as the visual encoder; while it can capture coarse global information, it often can miss fine-grained details that are relevant to the input query. To address these shortcomings, this work studies whether pre-trained text-to-image diffusion models can serve as instruction-aware visual encoders. Through an analysis of their internal representations, we find diffusion features are both rich in semantics and can encode strong image-text alignment. Moreover, we find that we can leverage text conditioning to focus the model on regions relevant to the input question. We then investigate how to align these features with large language models and uncover a leakage phenomenon, where the LLM can inadvertently recover information from the original diffusion prompt. We analyze the causes of this leakage and propose a mitigation strategy. Based on these insights, we explore a simple fusion strategy that utilizes both CLIP and conditional diffusion features. We evaluate our approach on both general VQA and specialized MLLM benchmarks, demonstrating the promise of diffusion models for visual understanding, particularly in vision-centric tasks that require spatial and compositional reasoning. Our project page can be found here . 1 Introduction Figure 1 : Overview. (Left) We present our full multimodal pipeline. Following LLaVA, we first extract visual features from the frozen diffusion model and pass the question as text-prompt. The LLM then uses these features to generate its answer. Cross-attention maps show that the model can use the question to focus on relevant regions (Right) We show examples on MMVP [ 1 ] where diffusion features outperform CLIP. Effective visual feature extraction remains a key challenge in designing robust multimodal large language models (MLLMs). Due to its vision-language pre-training, CLIP [ 2 ] is the de facto visual encoder for MLLM architectures, despite its inability to encode fine-grained visual details [ 1 , 3 , 4 ] and capture compositional information [ 5 , 4 ] . We contend that recent attempts to address these shortcomings [ 6 , 7 , 1 , 8 , 9 , 10 , 11 ] are flawed in their reliance on static visual features, which may not capture task-relevant visual details. For instance, answering a question about the visibility of a butterfly’s feet, as in Fig. 1 (right), requires precise localization and fine-grained semantic understanding. Although recent methods explore question-aware visual representations [ 12 , 13 ] , they often require substantial architectural changes to CLIP and employ fusion at a single stage thereby limiting image-query interactions [ 12 , 14 , 15 ] . In this work, we leverage text-to-image diffusion models as the visual encoder (pipeline shown in Fig. 1 (left)). These generative networks can create high-quality images that align well with the semantics and compositions described by a given text prompt [ 16 , 17 , 18 , 19 ] . Recent studies find that this is a result of their strong internal representations as well as the cross-attention mechanism, which is embedded throughout their architecture and modulates pixel features with the input text [ 20 , 21 , 22 , 23 ] . We first analyze unconditional features extracted from diffusion models across multiple blocks and timesteps. Although prior work has shown that such features can be repurposed for tasks like classification, segmentation, and depth estimation [ 24 , 25 , 26 , 27 ] , their utility for multimodal reasoning remains underexplored. We use the LLaVA framework for our investigation and inspect model performance using features across various blocks and timesteps. Our results on general-purpose and vision-centric benchmarks show that these features encode complementary information and, in many cases, match or outperform CLIP-based LLaVA models. We show examples in Fig 1 (right). Building on this, we inspect the text-conditional cross-attention maps. Fig. 1 (left) shows that these attention maps can focus on semantically relevant regions. We seek to quantify this alignment via image-text matching performance and find that diffusion cross-attention significantly outperforms CLIP in capturing spatial and compositional relationships (see Table 1 ). We also investigate how text conditioning modulates intermediate spatial features at different blocks and layers. While the impact is small for lower guidance values, amplifying this guidance can visibly change the structure of the features, with greater focus placed on image regions related to the text prompt. Furthermore, we discover a leakage effect where the language model can learn to extract the input caption fed to the diffusion model. We show how to quantify the effect of leakage and provide a mitigation strategy during training. We then turn our focus towards using questions as the text condition for the diffusion model. First, we demonstrate that using questions as text-input guides the intermediate features to highlight relevant regions. Based on our findings, we propose leveraging the complementary information in CLIP and diffusion features to build an improved multimodal pipeline and show encouraging performance. In summary, our contributions are as follows: • We show that unconditional diffusion features contain fine-grained structure and semantics, leading to improved performance on vision-centric tasks with +1.23% on BLINK-val. • We investigate what information is stored in text-conditioned cross-attention maps and find that they encode robust vision-language correspondence. We show that when used for image-text matching, these maps can match or exceed CLIP with a +5% gain on MMVP-VLM. • We provide comprehensive analysis to show how text conditioning modulates spatial features, enabling greater focus on query-specific regions. • We explore harnessing the complementary information in CLIP and conditional diffusion features and achieve promising results with a +6% improvement on MMVP over the LLaVA-v1.5-7B baseline. Figure 2 : Inspecting Diffusion Features . (a, b) We visualize spatial features for three image pairs from MMVP-VLM using PCA across blocks and timesteps. For (a), we use T=50; for (b), we fix U-L1-R1-B0-Cross-Q . We observe: (1) different blocks capture either shared semantics or image-specific details; (2) higher timesteps encode coarse layout, while lower timesteps emphasize fine-grained structure; and (3) features like out and res-out have tokens which can act as "registers" that act as shared global descriptors across similar images. (c) We plot average cosine similarity, finding that cross-q representations capture greater similarity compared to out features. 2 Related Work Vision Language Models Vision language modeling has been a popular topic with foundational image-text alignment papers such as [ 28 , 2 ] . Multimodal LLMs go one step further and have taken the success of large-scale pretrained LLMs and applied them to vision tasks [ 29 , 30 , 15 , 14 ] . However, they typically require a large amount of pre- or post-training to align the vision and language features. More recent methods like [ 31 , 32 , 33 ] show how visual instruction can be done quickly with low data while being competitive with strong baselines [ 34 , 15 ] across a wide variety of tasks. Numerous works and benchmarks have demonstrated the shortcomings of these methods, including their propensity for hallucination [ 35 , 36 , 37 ,