Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation

Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Hyunwoo Oh, Sungho Koh, Dong-Jin Kim · 2026

Read Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation by Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Hyunwoo Oh, Sungho Koh, Dong-Jin Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19158.txt.

Introduction

Lee and co-authors present a way to keep text-to-image models on target when the prompt names a rare concept. Their method blends the main prompt with helper anchor prompts at each step. A closed-form rule, drawn from Tweedie's identity, sets how much weight each prompt gets. The blend gives semantic support for rare ideas and shape support for edits. Tests on RareBench and FlowEdit beat fixed blends and prior training-free methods. The team reports gains in both meaning and structure.

Catalog record

CREATORKwanyoung Lee, SeungJu Cha, Yebin Ahn, Hyunwoo Oh, Sungho Koh, Dong-Jin Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73623
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19158

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Hyunwoo Oh, Sungho Koh, Dong-Jin Kim Abstract Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) — a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie’s identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines. Code is available here . 1 Introduction Text-to-image (T2I) diffusion models have achieved remarkable progress in generating photorealistic and semantically rich images from natural language descriptions 26 ; 22 ; 6 . Recent advances in large-scale training have yielded impressive fidelity across diverse visual domains and styles 2 ; 5 ; 39 . However, a persistent challenge remains: when the target concepts lie in low-density regions of the training distribution, these models often fail to produce faithful or structurally consistent results 28 . Figure 1 : When the target concept lies in a low-density region, the generated samples tend to drift toward semantically dominant, high-density concepts 33 in the learned score space, resulting in the suppression of rare or compositional attributes. Our proposed adaptive coefficient \gamma_{t}^{*} , derived from auxiliary prompt blending between \tilde{c}_{T} and its anchor \tilde{c}_{A} , dynamically corrects this bias and produces target-faithful results. Unlike a fixed coefficient \gamma_{t} , the adaptive \gamma_{t}^{*} adjusts per timestep to maintain a target-aligned denoising trajectory. This problem stems from the inherently long-tailed nature of text–image datasets, where common concepts (e.g., “frog”, “cat”) dominate, while rare or compositional ones (e.g., “hairy frog”, “origami cat”) appear sparsely or not at all 21 ; 28 . Consequently, the learned score function becomes under-constrained in these regions, causing generations to drift toward semantically dominant concepts 33 . This issue manifests across two related settings: (1) in rare concept generation 21 ; 28 , models fail to express target-specific attributes, and (2) in image editing 15 ; 37 ; 27 ; 19 , models struggle to preserve the original structure under uncommon or complex edits. To mitigate these failures, prior works leverage auxiliary prompts as stabilizing anchors—using semantically related frequent prompts for rare concept generation 21 and the original source prompt for image editing 15 . However, the key challenge lies in balancing the influence between the target prompt and its auxiliary anchor throughout the diffusion process. As shown in Fig. 1 , an improper balance leads to two failure modes: over-reliance on the anchor suppresses the target semantics, whereas insufficient anchoring yields unstable generations that deviate from the intended concept. This motivates the need for a principled formulation of how auxiliary and target prompts should be blended across timesteps. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) , a training-free framework that stabilizes diffusion generation by adaptively modulating the contribution of the auxiliary anchor throughout the denoising process. Prior methods, such as R2F 21 , address this trade-off by using fixed and heuristic prompt schedules that vary across prompts and tasks. In contrast, we derive a closed-form adaptive coefficient that dynamically balances the contributions of the target and anchor at each diffusion step, ensuring both semantic precision and structural stability. Building on Tweedie’s identity 25 , our goal is to align the posterior mean of the blended denoised estimate toward that of the target prompt. Then, we derive a closed-form adaptive coefficient that minimizes the distance between the posterior mean, ensuring the blended denoising trajectory remains aligned with the target concept rather than drifting toward high-density modes. This framework naturally unifies rare concept generation and image editing, both of which involve target concepts that lie in low-density regions, under a single adaptive optimization principle. We build intuition for our approach through a controlled toy example and a supporting proposition showing that the adaptive coefficient achieves a lower squared 2-Wasserstein distance than fixed interpolation under idealized conditions, followed by empirical validation on real-world tasks. We validate AAPB across two distinct tasks—rare concept generation on RareBench 21 and image editing on FlowEdit 15 dataset—and observe substantial improvements over prior training-free baselines, notably enhancing semantic accuracy for rare concepts and delivering more structure-preserving edits. The main contributions of this work are summarized as follows: 1. We introduce AAPB , a unified and training-free approach for target-faithful diffusion generation in low-density regions. 2. We derive a closed-form adaptive coefficient that replaces heuristic anchor scheduling by optimally balancing target and anchor guidance through Tweedie-based score alignment. 3. We provide theoretical insight via a controlled toy example and proposition showing that adaptive blending yields a lower squared 2-Wasserstein distance than fixed interpolation, and verify this behavior empirically. 4. We demonstrate strong and consistent gains on RareBench and FlowEdit datasets, improving semantic accuracy and structural fidelity across both rare concept generation and image editing tasks. 2 Related Works Rare Concept Generation. While large-scale pre-trained diffusion models 26 ; 22 ; 6 demonstrate strong visual quality, they still struggle to synthesize semantically rare or visually underrepresented concepts. SynGen 24 tackles fine-grained misalignment via cross-attention modulation but remains confined to frequent concepts seen during training. LLM-grounded diffusion frameworks 17 ; 36 ; 11 improve prompt understanding and compositional reasoning, yet still fail to generalize to tail concepts, causing unstable or inconsistent generations. Recent works, SeedSelect 28 and R2F 21 , leverage head or frequent samples to improve rare concept synthesis—SeedSelect in the image space by retrieving optimal noise seeds, and R2F in the semantic space via LLM-based frequent prompts. However, both rely on external references or fixed LLM schedules, limiting zero-shot generalization. Specifically, R2F alternates between target and anchor at the prompt level, preventing continuous score-space interpolation and per-timestep adaptivity. In contrast, we operate directly in score space, enabling closed-form adaptive coefficients to balance contributions per timestep. Image Editing. Training-free editing methods such as SDEdit 19 perturb an input image with Gaussian noise and denoise it via diffusion priors, but often suffer from structural degradation. Deterministic ODE inversion 27 improves reconstruction by reversing diffusion trajectories, yet demands fine discretization or prompt optimization for controllable edits. iRFDS 37 leverages rectified flow distillation to achieve inversion and editing without stochastic noise, but struggles with local attribute consistency. FlowEdit 15 employs an inversion-free ODE that effectively maintains structural consistency, although it may lose original content structure under stronger or more complex edit instructions. Building on this line, our AAPB framework targets zero-shot semantic

Read the complete text