ISAC: Training-Free Instance-to-Semantic Attention Control for Improving Multi-Instance Generation

Sanghyun Jo [email protected], Wooyeol Lee [email protected], Ziseok Lee [email protected], Kyungsu Kim [email protected] · 2025

Read ISAC: Training-Free Instance-to-Semantic Attention Control for Improving Multi-Instance Generation by Sanghyun Jo [email protected], Wooyeol Lee [email protected], Ziseok Lee [email protected], Kyungsu Kim [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20935.txt.

Introduction

Sanghyun Jo and co-authors fix a common flaw in AI image generators in a 2025 paper. When a prompt asks for several objects, models often drop some, merge them, or mix their traits. The team traces the trouble to the earliest cleanup steps, before object borders settle. Their method, called ISAC, needs no retraining.

ISAC first steadies the layout that self-attention sketches, then ties word meanings inside each region. Across three test suites it beat other training-free fixes. It also sharpened rough boxes from layout tools into clean object masks. Readers get a plug-in upgrade for crowded image prompts.

Catalog record

CREATORSanghyun Jo [email protected], Wooyeol Lee [email protected], Ziseok Lee [email protected], Kyungsu Kim [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20935

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ISAC: Training-Free Instance-to-Semantic Attention Control for Improving Multi-Instance Generation Sanghyun Jo [email protected], Wooyeol Lee [email protected], Ziseok Lee [email protected], Kyungsu Kim [email protected] Abstract Text-to-image diffusion models excel at generating single-instance scenes but struggle with multi-instance scenarios, often merging or omitting objects. Unlike previous training-free approaches that rely solely on semantic-level guidance without addressing instance individuation, our training-free method, I nstance-to- S emantic A ttention C ontrol (ISAC), explicitly resolves incomplete instance formation and semantic entanglement through an instance-first modeling approach. This enables ISAC to effectively leverage a hierarchical, tree-structured prompt mechanism, disentangling multiple object instances and individually aligning them with their corresponding semantic labels. Without employing any external models, ISAC achieves up to 52% average multi-class accuracy and 83% average multi-instance accuracy by effectively forming disentangled instances. The code will be made available upon publication. † † firstpage: 1 † † editor: My editor keywords text-to-image generation, attention control, multi-instance generation, training-free, diffusion model 1 Introduction Text-to-image (T2I) diffusion models ( rombach2022high ; podell2023sdxl ; esser2024scaling ; chen2023pixartalpha ; flux2024 ) have demonstrated remarkable capabilities in generating high-quality images from textual descriptions. While these models excel at generating single objects, they often struggle when asked to create images with multiple distinct objects. For example, when prompted with “A photo of two cats and a dog”, even advanced models like SDXL ( podell2023sdxl ) and SD3.5-M ( esser2024scaling ) frequently fail in two common ways, as shown in Figure 1 : (1) instance missing , where not all requested objects appear in the image, and (2) instance merging , where different objects’ features blend together into a single entity. Previous research has shown that cross-attention maps within the model’s UNet play a key role in determining object placement ( hertz2022prompt ) . Based on this insight, several researchers have developed methods to manipulate these cross-attention maps ( feng2023trainingfree ; Chen_2024_TEBOpt ; chefer2023attendandexcite ; rassin2024linguistic ; guo2024initno ; hu2024token ) . Some methods boost attention to neglected words ( chefer2023attendandexcite ; guo2024initno ) , while others try to separate attention maps for different objects ( hu2024token ; rassin2024linguistic ; izadi2025fine ) . However, these approaches still struggle because they focus only on semantic features without properly understanding object shapes and boundaries. Our investigations show that even when strictly enforcing separation between attention maps using maximum pixel-wise overlap (MPO) loss ( 6 ), the results often include fragmented or unnaturally divided objects. As Figure 7 illustrates, this occurs because cross-attention maps primarily capture what objects are (semantics) but not their physical shapes and boundaries. In this paper, we introduce I nstance-to- S emantic A ttention C ontrol (ISAC). While existing methods focus solely on semantic-level guidance without first resolving instance formation, ISAC explicitly addresses incomplete instance formation and multi-semantic entanglement through an instance-first generation approach. Our key insight is that self-attention maps within the diffusion model, especially in early generation steps, naturally distinguish between different object shapes. ISAC leverages this by first establishing distinct object instances using self-attention maps, and then guiding semantic assignment through cross-attention maps. This two-phase approach enables our framework to support a hierarchical, tree-structured prompt mechanism, where multiple object instances are first disentangled and then individually aligned with their corresponding semantics. In summary, our main contributions are: • We identify that prior methods fail by focusing on semantic guidance without first resolving instance individuation, and propose a solution that explicitly models instance formation before semantic assignment. • ISAC introduces a hierarchical, tree-structured approach that first disentangles object instances using self-attention maps, then aligns each instance with its corresponding semantics through instance-aware semantic attention. • Our method significantly outperforms existing approaches across multiple model architectures (SD1.5 ( rombach2022high ) , SDXL ( podell2023sdxl ) , SD3.5-M ( esser2024scaling ) ), achieving up to 52% average multi-class accuracy and 83% average multi-instance accuracy by effectively forming disentangled instances. Figure 1 : Comparison between existing text-to-image diffusion models ( rombach2022high ; podell2023sdxl ; esser2024scaling ) (left) and the proposed ISAC framework (right) on the prompt "A photo of two cats and a dog". Current methods (SD1.5 rombach2022high , SDXL podell2023sdxl , SD3.5-M esser2024scaling ) struggle to clearly separate individual instances, leading to merged or overlapped objects (red dashed circles). In contrast, our ISAC explicitly utilizes early-stage self-attention to accurately identify distinct, non-overlapping instances (green dashed circles) and assigns them precise semantic labels, resulting in clear class boundaries without additional training or supervision. 2 Related Work Cross-Attention Manipulation for Semantic Control. The pivotal work of hertz2022prompt established cross-attention maps as the primary interface between text semantics and spatial layouts in diffusion models. This insight spawned two dominant strategies for multi-instance generation: attention enhancement ( chefer2023attendandexcite ; feng2023trainingfree ; Chen_2024_TEBOpt ; guo2024initno ) and attention separation ( rassin2024linguistic ; izadi2025fine ; hu2024token ; meral2024conform ) . Attend-and-Excite ( chefer2023attendandexcite ) pioneered the former approach by maximizing attention response peaks for neglected objects, this often creates overlapping regions that lead to instance merging. Subsequent methods like InitNO ( guo2024initno ) introduced noise optimization to strengthen cross-attention signals but remained constrained by their focus on localized features rather than holistic shapes. TEBOpt ( Chen_2024_TEBOpt ) shows interesting perspective by optimizing CLIP ( radford2021learning ) text embeddings via strengthening single instance embedding signals. The separation paradigm, exemplified by SynGen ( rassin2024linguistic ) , employs syntactic analysis and KL-divergence losses to minimize overlap between unrelated entities’ attention maps. However, our experiments in Figure 7 and Table 3 reveal these global distribution metrics, such as KL-divergence and Intersection-of-Union (IoU), fail to enforce the pixel-level discontinuities needed for distinct instances. Self-Attention for Structural Guidance. While most methods focus on cross-attention, recent work hints at self-attention’s untapped potential. Particularly relevant to multi-instance scenarios, jo2025diffegg observed that early-step self-attention maps excel at decoupling distinct instance shapes, while cross-attention maps maintain a primary focus on semantic features throughout various stages. InitNO ( guo2024initno ) briefly explores self-attention peaks but lacks ISAC’s systematic framework for converting these signals into coherent instance masks. Figure 2 illustrates this observation, showing that self-attention maps in early diffusion steps effectively delineate distinct instance shapes, finalizing the instance layout then the capability diminishes in later steps. It is also noteworthy that some approaches, such as ( bansal2023universal ; kang2025counting ) , propose using pretrained vision models to enhance multi-instance generation. However, these models, such as Grounding DINO ( liu2024grounding ) , often struggle in the early diffusion steps where semantic information is still nascent, highlighting the strength of ISAC’s intrinsic signal. Decoupling Instance and Semantic Refinement. The dichotomy between structural formation and semantic assignment mirrors findings in diffusion process analysis. ( lee2025beta ) established through Fourier analysis that low-frequency structural components emerge early, while high-frequency details develop later. ISAC operationalizes this insight by aligning self-attention guidance in early diffusion steps and cross-attention refinement in later diffusion steps, avoiding the conflicting objectives that plague single-stage approaches. Figure 2 : Instance-to-semantic dynamics in diffusion models. Early diffusion steps prioritize instance decoupling, where self-attention maps begin to delineate distinct shapes, while later steps refine semantic details. The figure also illustrates that semantic features of denoised image are nascent in early stages, making guidance from pretrained vision models such as Grounding DINO ( liu2024grounding ) challenging until semantics become clearer. We used SD3.5-M (

Read the complete text