TIDE: Achieving Balanced Subject-Driven Image Generation via Target-Instructed Diffusion Enhancement Volume: 44 Issue: 7

Jibai Lin 1,2,3 Bo Ma 1,2,3 Yating Yang 1,2,3 Xi Zhou 1,2,3 Rong Ma 1,2,3 Turghun Osman 1,2,3 Ahtamjan Ahmat 1,2,3 Rui Dong 1,2,3 Lei Wang 1,2,3 1 Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Xinjiang Laboratory of Min · 2025

Read TIDE: Achieving Balanced Subject-Driven Image Generation via Target-Instructed Diffusion Enhancement Volume: 44 Issue: 7 by Jibai Lin 1,2,3 Bo Ma 1,2,3 Yating Yang 1,2,3 Xi Zhou 1,2,3 Rong Ma 1,2,3 Turghun Osman 1,2,3 Ahtamjan Ahmat 1,2,3 Rui Dong 1,2,3 Lei Wang 1,2,3 1 Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Xinjiang Laboratory of Min (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06499.txt.

Introduction

Keeping a pictured subject looking like itself while following an edit instruction is the tension explored by Jibai Lin and eight co-authors in a 2025 image generation paper. Subject-driven generation changes a specific person or object in an image as text directs. Older methods often sacrifice either identity or obedience to the instruction.

Their framework, TIDE, learns from triples of a reference image, an instruction, and target images. Some targets balance preservation and compliance well; others are distorted. Training favors the winning targets over the losing ones, a form of preference learning that needs no extra tuning when the model is used. Tests on standard benchmarks show more faithful subjects and better instruction following than rivals. The same approach also works for structure-guided generation, image-to-image edits, and blending between text and image.

Catalog record

CREATORJibai Lin 1,2,3 Bo Ma 1,2,3 Yating Yang 1,2,3 Xi Zhou 1,2,3 Rong Ma 1,2,3 Turghun Osman 1,2,3 Ahtamjan Ahmat 1,2,3 Rui Dong 1,2,3 Lei Wang 1,2,3 1 Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Xinjiang Laboratory of Min
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06499

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TIDE: Achieving Balanced Subject-Driven Image Generation via Target-Instructed Diffusion Enhancement Volume: 44 Issue: 7 Jibai Lin 1,2,3 Bo Ma 1,2,3 Yating Yang 1,2,3 Xi Zhou 1,2,3 Rong Ma 1,2,3 Turghun Osman 1,2,3 Ahtamjan Ahmat 1,2,3 Rui Dong 1,2,3 Lei Wang 1,2,3 1 Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3 Xinjiang Laboratory of Minority Speech and Language Information Processing Abstract Subject-driven image generation (SDIG) aims to manipulate specific subjects within images while adhering to textual instructions, a task crucial for advancing text-to-image diffusion models. SDIG requires reconciling the tension between maintaining subject identity and complying with dynamic edit instructions, a challenge inadequately addressed by existing methods. In this paper, we introduce the Target-Instructed Diffusion Enhancing (TIDE) framework, which resolves this tension through target supervision and preference learning without test-time fine-tuning. TIDE pioneers target-supervised triplet alignment, modelling subject adaptation dynamics using a (reference image, instruction, target images) triplet. This approach leverages the Direct Subject Diffusion (DSD) objective, training the model with paired "winning" (balanced preservation-compliance) and "losing" (distorted) targets, systematically generated and evaluated via quantitative metrics. This enables implicit reward modelling for optimal preservation-compliance balance. Experimental results on standard benchmarks demonstrate TIDE’s superior performance in generating subject-faithful outputs while maintaining instruction compliance, outperforming baseline methods across multiple quantitative metrics. TIDE’s versatility is further evidenced by its successful application to diverse tasks, including structural-conditioned generation, image-to-image generation, and text-image interpolation. Our code is available at https://github.com/KomJay520/TIDE . \SpecialIssueSubmission keywords Subject-driven image generation, Diffusion models, Target supervision, Preference learning, Triplet alignment † † year: 2025 † † editors: 2025 † † subject: sss † † orcid: 0009-0001-4722-4461 † † orcid: 0000-0003-3082-648X † † orcid: 0000-0002-2639-3944 † † teaser: Our TIDE framework achieves high-fidelity subject-driven generation from single reference images via target-supervised learning. Our model also supports diverse tasks like structure-controlled generation, image-to-image generation, and interpolation between reference images and textual prompts to demonstrate its versatility in subject manipulation. 1 Introduction Subject-Driven image generation (SDIG) aims to synthesize images that preserve the identity of a specified subject (e.g., a cat, object) while adapting to textual editing instructions (e.g., "The cat* swimming in a pool") [ Gal et al.(2022) , Ruiz et al.(2023) , Kumari et al.(2023) , Li et al.(2023) , Zhang et al.(2024) ] . This task extends text-to-image diffusion models [ Dhariwal and Nichol(2021) , Rombach et al.(2022) , Ahn et al.(2024) , Chefer et al.(2024) ] to enable controlled subject manipulation, critically expanding their utility for realistic applications like personalized content creation and visual storytelling. At its core, SDIG demands precise reconciliation of two competing objectives: invariant preservation of subject identity against dynamic compliance with edit instructions. This process advances controllable generation techniques for real-world subject editing scenarios. Existing SDIG methods can be broadly classified into fine-tuning based and fine-tuning free methods, with the key distinction lying in whether they perform parameter updates during test time. On the one hand, fine-tuning based methods fine-tune pre-trained diffusion model or its components using 3-5 reference images at test-time [ Ruiz et al.(2023) , Avrahami et al.(2023) , Hyung et al.(2024) ] . These methods embed subject features into a placeholder token (e.g., [V]) and combine it with text prompts (e.g., The [V] cat swimming in a pool) to generate subject-driven images. However, they suffer from catastrophic forgetting: retraining on novel subjects leads to the overwriting of previously learned representations limiting their scalability to diverse subject domains. On the other hand, fine-tuning free methods preserve parameters of base model by encoding reference images via external modules (CLIP/BLIP) and injecting subject features through trainable adapters [ Li et al.(2023) , Zhang et al.(2024) , Ma et al.(2024) ] . Yet, their self-supervised reconstruction paradigm, as shown in the top row of Figure 1 , is forced to use reference images as pseudo-targets. Due to missing instruction-edited pairs, they conflate preservation (e.g., retaining object appearance) and modification (e.g.,altering background). This ambiguity prevents models from learning the critical balance between feature retention and semantic adaptation. In summary, existing approaches fail to inadequate balance subject preservation and instruction compliance: fine-tuning based methods sacrifice generalization for subject fidelity, while fine-tuning free methods lack explicit supervision for edit disentanglement. Figure 1: Different of Self-supervised Reconstruction and Target-supervised Learning in subject-driven image generalization. Self-supervised Reconstruction always use the input images as label images, result to bad output in applications. To resolve the preservation-compliance trade-off, we propose TIDE (Target-Instructed Diffusion Enhancing), a fine-tuning-free framework built upon two key innovations: explicit triplet supervision and the Direct Subject Diffusion (DSD) objective. First, as demonstrated in the middle row of Figure 1 , TIDE introduces reference-instruction-target triplet supervision, which is a structured guidance mechanism absent in prior self-supervised methods. Unlike approaches that rely solely on reference images as targets (top row), the target images in triplet explicitly encode desired attribute edits (e.g., pose, background changes), enabling the model to learn fine-grained mappings between instructions and subject adaptations. Crucially, TIDE proposes the DSD objective tailored for SDIG. Specifically, DSD adapts the pairwise preference learning to image generation by integrating subject-specific features from reference images, enabling the model to learn from paired "winning"/"losing" target images. For each instruction (e.g., "Cat swimming in a pool"), the preference pairs which encode human-like quality judgments for balanced outputs, train the model to prefer "winning" targets (balanced outputs) over "losing" targets (unbalanced, e.g., instruction or feature loss). This preference signal allows TIDE to implicitly model a reward function that guides the diffusion process to generate semantically consistent images while preserving critical subject attributes. As visualized in Figure 1 bottom-row comparisons, TIDE generates images faithfully following instructions while preserving subject identity. To implement these innovations, we adapt prior multimodal fusion techniques through an Image Projection Module with Image Cross-Attention Module (IPM-ICAM). This module encodes reference features to condition the diffusion process. Extensive experiments have shown that our method with this lightweight module achieves comparable results to other methods in subject-driven image generation. To summarize, our contributions are listed as follows: • We propose a novel framework, named TIDE, for personalized subject-driven image generation, which uses triplet data to explicitly specify feature preservation and modification rules. • In TIDE, we propose Direct Subject Diffusion, a novel preference learning objective that enables implicit optimization of both subject fidelity and instruction compliance. • Our extensive experiments have validated the robustness and flexibility of our approach, showcasing its capability to deliver competitive performance among SDIG methods. It also supports diverse tasks including structural-conditioned generation, image-to-image generation, and text-image interpolation. 2 Related Work 2.1 Aligning Large Language Models LLMs [ Achiam et al.(2023) , Ye et al.(2024) ] are typically aligned with human preferences through supervised fine-tuning (SFT) [ Dong et al.(2023) ] on demonstration data followed by reinforcement learning from human feedback (RLHF) [ Bai et al.(2022) ] . RLHF learns a reward model from pairwise output comparisons to guide policy optimization via policy-gradient methods [ Mnih et al.(2016) ] , but faces notable challenges including high computational costs from nested training loops, sensitivity to hyper-parameter choices, and susceptibility to reward hacking [ Dubois et al.(2023) , Skalse et al.(2022) ] . Recent approaches mitigate these issues through reward-weighted supervised learning, ranking loss minimization and

Read the complete text