ADAPT: A ttention D riven A daptive P rompt Scheduling and In T erpolating Orthogonal Complements for Rare Concepts Generation
Kwanyoung Lee, Hyunwoo Oh, SeungJu Cha, Sungho Koh, Dong-Jin Kim · 2026
Read ADAPT: A ttention D riven A daptive P rompt Scheduling and In T erpolating Orthogonal Complements for Rare Concepts Generation by Kwanyoung Lee, Hyunwoo Oh, SeungJu Cha, Sungho Koh, Dong-Jin Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19157.txt.
Introduction
Lee and co-authors help image models draw rare mixes of traits that they seldom saw in training. Their ADAPT method plans the prompt steps in a fixed way, guided by attention scores. It blends in parts of the prompt that sit at right angles to the main text cues. No new training is needed. On the RareBench test set, ADAPT tops prior methods for rare mixes. The team reports clear gains while image quality stays intact.
Catalog record
| CREATOR | Kwanyoung Lee, Hyunwoo Oh, SeungJu Cha, Sungho Koh, Dong-Jin Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52228 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19157
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ADAPT: A ttention D riven A daptive P rompt Scheduling and In T erpolating Orthogonal Complements for Rare Concepts Generation Kwanyoung Lee, Hyunwoo Oh, SeungJu Cha, Sungho Koh, Dong-Jin Kim Abstract Generating rare compositional concepts in text-to-image synthesis remains a challenge for diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for prompt scheduling, they suffer from inherent variance due to the randomness of language models and suboptimal guidance from iterative text embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity. Code is available here . Figure 1 : Qualitative comparison between R2F and ADAPT (ours) across rare concepts. Our proposed method significantly enhances R2F in a zero-shot way, demonstrating superior capability in text-image alignment. All the pairs are generated with the same seed (42). 1 Introduction The generation of rare compositional images has become increasingly important as text-to-image models are widely used to create such compositions 11 . However, generating rare compositional concepts in text-to-image synthesis remains challenging for diffusion models 6 ; 20 ; 16 , particularly for attributes that are uncommon or absent in training data 21 . Furthermore, existing attribute binding methods 19 ; 3 cannot accurately bind a rare attribute to the common object, such as “a banana-shaped car” and “a black and white checkerboard crocodile.” R2F 15 addresses this by leveraging GPT-4o 10 for concept mapping to generate auxiliary frequent prompts and to determine visual detail levels, which linearly map to scheduling stop points between rare and frequent concepts during generation. However, R2F’s dependence on GPT-4o induces variance in the created prompts and visual detail levels due to the inherent randomness of the language model. Additionally, linear mapping from visual detail levels to stop points is heuristic and misaligned with token-level semantic information during generation. Furthermore, R2F iteratively switches text embeddings between rare and frequent prompts, making it difficult to provide semantically precise and consistent guidance during the generation process. In this work, our goal is to utilize deterministic prompt schedules with semantically precise and consistent guidance through denoising steps for generating rare compositions. To address the problem of variance, we propose Adaptive Prompt Scheduling (APS) to mitigate GPT-4o variance on visual detail levels and semantically align prompt scheduling with text tokens. In particular, we leverage max spatial attention scores to measure attention activation for each token during generation 3 . Our key insight leverages spatial attention convergence 27 as a principled indicator of semantic feature saturation, thereby enabling semantically aligned determination of optimal stop points for prompt scheduling. Second, to improve guidance for rare and frequent concepts, we propose Pooled Embedding Manipulation (PEM) , which consistently applies a merged pooled embedding between rare and frequent prompts, rather than iteratively switching between them. When merging rare and frequent prompts, we propose to project the rare pooled text embedding onto the orthogonal complement of the frequent prompt, thereby extracting a direction that disentangles rare-specific semantics. We then interpolate between frequent and rare-specific semantics 5 , but uniform interpolation strength results in over-suppressing base semantics or under-emphasizing rare attributes. To address this, we introduce an adaptive weighting strategy 23 that determines the interpolation scale based on the cosine similarity of CLIP’s pooled embedding space. This adaptive scale is then used for linear interpolation along the projection directions, yielding modulated pooled embeddings that balance base semantic preservation with the enhancement of rare attributes. Lastly, some prompts exhibit substantial semantic differences between rare and frequent concepts (e.g., “A metallic humanoid figure” and “A clown made of steel”), making attribute-specific manipulation challenging. Therefore, we extract the attribute text (e.g., “made of steel” in “A clown made of steel”) by modifying R2F’s concept mapping instructions for LLM and introduce Latent Space Manipulation (LSM) to extract disentangled guidance from the attribute text and apply it to the model within attention layers through an orthogonal guidance vector with a tunable scaling factor. Through these contributions, we propose a framework that improves guidance with rare and frequent concepts, ADAPT , for generating rare compositional concepts: Our contributions are summarized as follows: • We propose APS , which leverages attention scores to determine optimal stop points for prompt scheduling, thereby eliminating GPT-4o dependency in determining stop points and achieving semantic alignment. • We introduce a dual-level embedding manipulation framework that combines PEM for precise and consistent guidance on rare semantics and LSM for attribute-specific guidance, enabling disentangled directional control over rare concept generation. • Through comprehensive experiments, we demonstrate that our framework achieves superior performance in accurately reflecting rare attribute semantic information during the generation process. 2 Related Work 2.1 Enhancing Text-Alignment in Diffusion Models Recently, diffusion models 20 ; 16 ; 4 ; 6 have demonstrated impressive capabilities in generating photorealistic images conditioned on diverse textual inputs. However, they often struggle to fully comprehend complex text prompts, leading to suboptimal or misaligned generations 19 . To improve text-image alignment, various strategies have been proposed, including the use of additional conditions 12 ; 28 ; 24 , LLM-based instruction generation 7 ; 26 ; 14 , and cross-attention information 19 ; 3 ; 2 . For example, 28 ; 12 ; 24 employ spatial conditions such as bounding boxes to guide the layout according to the text. LLM instruction-based approaches 7 ; 26 ; 14 utilize textual refinements or additional context generated by large language models to improve semantic understanding. Another line of work 2 ; 19 ; 3 leverages cross-attention maps to extract token-wise alignments and enhance fine-grained semantic consistency. While these methods have addressed various text alignment problems, they primarily focus on generalizable and intuitive cases—those that are relatively easy for humans to imagine or interpret. 2.2 Rare Concept Generation To better utilize diffusion models in creative fields, some users have expressed the need to generate rare concept images that may not exist in the real world 11 . Although methods like SynGen 19 and Attend \& Excite 3 can generate reasonably accurate images for certain rare concepts, they still struggle when the target concept is extremely rare (e.g., “A horned bearded spotted raccoon smiling”) 15 . To address this issue, researchers have proposed more generalized approaches 21 ; 15 . For example, 21 shows that rare concepts can be successfully generated by carefully selecting generation seeds in the noise space, using a small set of reference images. More recently, R2F 15 improves the ability of diffusion models to generate rare compositional concepts in a training-free manner by leveraging responses from LLMs. Specifically, they construct an auxiliary frequent prompt from a rare concept and use it iteratively as input, guided by GPT-4o 10 responses. However, the inherent response variance of GPT-4o causes significant variation