Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers

Chaehyun Kim, Heeseong Shin, Eunbeen Hong, Heeji Yoon, Anurag Arnab, Paul Hongsuck Seo, Sunghwan Hong, Seungryong Kim · 2025

Read Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers by Chaehyun Kim, Heeseong Shin, Eunbeen Hong, Heeji Yoon, Anurag Arnab, Paul Hongsuck Seo, Sunghwan Hong, Seungryong Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18096.txt.

Introduction

Chaehyun Kim and 7 co-authors open up text-to-image models to find where word sense lands in a picture. Their 2025 frame, Seg4Diff, studies joint attention in a diffusion transformer that holds image and text tokens side by side. Such maps have been little mapped, layer by layer.

One block stands out as a grounding expert: it links text tokens to neat image zones, again and again. Those links yield good masks for what each word names. A light fine-tune with mask-tagged images sharpens that skill. Image quality rose too, which shows the grouping is a real trait of these models.

Catalog record

CREATORChaehyun Kim, Heeseong Shin, Eunbeen Hong, Heeji Yoon, Anurag Arnab, Paul Hongsuck Seo, Sunghwan Hong, Seungryong Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18096

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers Chaehyun Kim, Heeseong Shin, Eunbeen Hong, Heeji Yoon, Anurag Arnab, Paul Hongsuck Seo, Sunghwan Hong, Seungryong Kim Abstract Text-to-image diffusion models excel at translating language prompts into photorealistic images by implicitly grounding textual concepts through their cross-modal attention mechanisms. Recent multi-modal diffusion transformers extend this by introducing joint self-attention over concatenated image and text tokens, enabling richer and more scalable cross-modal alignment. However, a detailed understanding of how and where these attention maps contribute to image generation remains limited. In this paper, we introduce Seg4Diff ( Seg mentation for Diff usion), a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image. Through comprehensive analysis, we identify a semantic grounding expert layer, a specific MM-DiT block that consistently aligns text tokens with spatially coherent image regions, naturally producing high-quality semantic segmentation masks. We further demonstrate that applying a lightweight fine-tuning scheme with mask-annotated image data enhances the semantic grouping capabilities of these layers and thereby improves both segmentation performance and generated image fidelity. Our findings demonstrate that semantic grouping is an emergent property of diffusion transformers and can be selectively amplified to advance both segmentation and generation performance, paving the way for unified models that bridge visual perception and generation. Figure 1 : We introduce Seg4Diff , a systematic framework designed to analyze and enhance the emergent semantic grounding capabilities of multi-modal diffusion transformer (MM-DiT) blocks in text-to-image diffusion transformers (DiTs). Here, semantic grounding expert refers to a specific MM-DiT block responsible for establishing semantic alignment between text and image features. 1 Introduction The emergence of text-to-image (T2I) diffusion models have revolutionized visual content creation by allowing users to generate photorealistic images from natural-language prompts rombach2022high ; ho2020denoising ; song2020denoising . The success of these models imply that text-to-image diffusion models are highly capable of associating visual and textual representations, as the generated images accurately reflect the given textual prompt. In this regard, a growing body of work shows that this semantic grounding is encoded in the cross-modal attention maps of the diffusion models hertz2022prompt ; nichol2021glide ; saharia2022photorealistic ; podell2023sdxl , which have already been actively exploited for generative downstream tasks such as image editing, inpainting and personalization ruiz2023dreambooth ; hertz2022prompt ; brooks2023instructpix2pix ; mokady2022nulltext . Furthermore, several studies wang2024diffusionmodelsecretlytrainingfree ; sun2024iseg show that the cross-attention maps can be used to perform visual perception tasks such as open-vocabulary semantic segmentation, revealing that diffusion models can double as training-free segmentation models. However, the attention map produced by the standard U-Net-based ronneberger2015u ; rombach2022high models is often noisy and spatially fragmented cai2025freemask , limiting the fidelity of the resulting masks when leveraged without heuristic refinement. Recent advances replace the U-Net architecture with diffusion transformers (DiTs) peebles2023scalable , which benefits from the strong representation power and scalability of transformers for achieving state-of-the-art image generation quality. In particular, the multi-modal diffusion transformer (MM-DiT) esser2024scaling ; sd35 ; flux2024 introduces multi-modal attention, which concatenates text and image tokens and applies joint self-attention, enabling richer cross-modal interaction. This encourages us to explore the internal representations and the characteristics of the MM-DiT-based models. However, as opposed to U-Net based models, which have been extensively studied meng2024not , DiT-based models and their characteristics remain underexplored, in which a detailed understanding of how and where these attention maps contribute to image generation remains limited. In this paper, we first conduct an in-depth analysis of the joint attention mechanism in MM-DiT models. We characterize the distribution of attention scores to discover active cross-modal interaction, and complement this with feature similarity measure and norm analysis to assess which modality—textual or visual—exerts greater influence on the output representations. Together, these perspectives isolate a small subset of layers that consistently align textual semantics with contiguous image regions. These observations motivate us to further analyze this emergent semantic grounding capability. Subsequently, we propose an open-vocabulary segmentation scheme leveraging MM-DiT, which demonstrates that the identified layer, semantic grounding expert , yields high-quality segmentation masks, confirming their inherently competitive capability for open-vocabulary semantic segmentation. Extended analysis reveals that these attention map is in fact further decomposed into attention heads, which capture distinct parts of the semantic region and finally sum up to construct a complete semantic mask. Moreover, without explicit class information, we reveal that <pad> tokens can decompose the image into meaningful semantic regions, acting as anchors for the unconditional generation. Building on these findings, we introduce a lightweight fine-tuning scheme, called mask alignment for segmentation and generation (MAGNET) , that explicitly strengthens semantic grouping by leveraging mask-annotated image data in selected MM-DiT layers. Our primary motivation is to reinforce the emergent segmentation capability of these layers, enabling more accurate open-vocabulary semantic masks. Interestingly, this targeted adaptation also yields a modest but consistent improvement in image synthesis quality as a by-product of enhancing the cross-modal semantic alignment. Taken together, our stepwise analysis and selective refinement establish semantic grouping as a salient property of diffusion transformers for text-conditioned image generation—one that can be leveraged with minimal compute to advance both dense recognition and generative fidelity. These results illuminate the internal dynamics of diffusion transformers and point toward unified models that excel at both generation and perception. In summary, our contributions are as follows: • We provide in-depth analysis and investigation of learned representations of MM-DiT within its multi-modal attention layers. • We identify critical layers in MM-DiT that are essential for preserving text-conditioned semantics throughout the generation process, which reveals strong semantic grounding. • We demonstrate a zero-shot segmentation scheme to extract segmentation masks from the specific layers, and further enhance generation quality by enhancing their localization capability. 2 Related Work Emergent properties of diffusion models. Diffusion models, which generate images by iteratively denoising Gaussian noise, have transformed generative modeling ho2020denoising ; song2020denoising . U-Net-based text-to-image models rombach2022high surpassed GANs goodfellow2020generative , while recent transformer backbones peebles2023scalable further raised the bar. In particular, multi-modal diffusion transformer (MM-DiT) esser2024scaling , achieve stronger scalability and semantic alignment by fusing image and text tokens through joint attention. Beyond generation, pretrained diffusion models have been adapted to perception tasks such as correspondence zhang2023tale ; tang2023emergent , segmentation wang2024semflow , tracking luo2024diffusiontrack , depth estimation ke2024repurposing ; hu2024depthcrafter and video understanding velez2025image . On the other hand, inspired by Prompt-to-Prompt hertz2022prompt , which first proposed to leverage the cross-modal attention maps of text-to-image diffusion models revealing that text information is delicately aggregated into image generation via the attention mechanism, a rich line of works explored attention maps in different areas, such as image editing cao2023masactrl and conditional generation ahn2024self . Building on this, we focus specifically on analyzing the joint attention mechanism of MM-DiT, uncovering its rich semantic structure and its potential for segmentation. This perspective not only bridges generation and perception but also highlights attention as a central building block for semantic reasoning in diffusion models. Leveraging diffusion models for perception tasks. Recently, text-to-image diffusion models rombach2022high ; podell2023sdxl ; esser2024scaling have been adapted beyond generation to tackle discriminative vision tasks with remarkable success. For segmentation, ODISE xu2023open integrates a frozen diffusion backbone with CLIP, while adapter-based methods zhao2023unleashing leverage cross-attention for referring segmentation. For geometry, lightweight fine-tuning

Read the complete text