Steering Guidance for Personalized Text-to-Image Diffusion Models
Sunghyun Park, Seokeon ChoiHyoungwoo ParkSungrack Yun, Qualcomm AI Research {sunpar, seokchoi, hwoopark, sungrack}@qti.qualcomm.com · 2025
Read Steering Guidance for Personalized Text-to-Image Diffusion Models by Sunghyun Park, Seokeon ChoiHyoungwoo ParkSungrack Yun, Qualcomm AI Research {sunpar, seokchoi, hwoopark, sungrack}@qti.qualcomm.com (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00319.txt.
Introduction
Sunghyun Park, Seokeon Choi, Hyoungwoo Park, and Sungrack Yun wrote this computer vision paper, published in two thousand twenty-five, on steering a personalized image model at sampling time. Fine-tuning a generator on a few photos of a subject trades off against the model's grasp of text: push too hard and prompts stop working.
Standard guidance tricks cannot find the middle ground, the authors argue. Their personalization guidance uses a deliberately unlearned weak model, conditioned on an empty prompt, as a counterweight. Sliding a mix between the base and fine-tuned weights during sampling sets how strong the unlearning pull is. That steers outputs toward a balanced space where the subject looks right and the text still lands. The method adds no compute cost and plugs into many fine-tuning schemes. Experiments show gains on both fidelity and text alignment.
Catalog record
| CREATOR | Sunghyun Park, Seokeon ChoiHyoungwoo ParkSungrack Yun, Qualcomm AI Research {sunpar, seokchoi, hwoopark, sungrack}@qti.qualcomm.com |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-00319
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Steering Guidance for Personalized Text-to-Image Diffusion Models Sunghyun Park, Seokeon ChoiHyoungwoo ParkSungrack Yun, Qualcomm AI Research {sunpar, seokchoi, hwoopark, sungrack}@qti.qualcomm.com Abstract Personalizing text-to-image diffusion models is crucial for adapting the pre-trained models to specific target concepts, enabling diverse image generation. However, fine-tuning with few images introduces an inherent trade-off between aligning with the target distribution (e.g., subject fidelity) and preserving the broad knowledge of the original model (e.g., text editability). Existing sampling guidance methods, such as classifier-free guidance (CFG) and autoguidance (AG), fail to effectively guide the output toward well-balanced space: CFG restricts the adaptation to the target distribution, while AG compromises text alignment. To address these limitations, we propose personalization guidance, a simple yet effective method leveraging an unlearned weak model conditioned on a null text prompt. Moreover, our method dynamically controls the extent of unlearning in a weak model through weight interpolation between pre-trained and fine-tuned models during inference. Unlike existing guidance methods, which depend solely on guidance scales, our method explicitly steers the outputs toward a balanced latent space without additional computational overhead. Experimental results demonstrate that our proposed guidance can improve text alignment and target distribution fidelity, integrating seamlessly with various fine-tuning strategies. † † ∗ These two authors contributed equally to this work. † † † Qualcomm AI Research is an initiative of Qualcomm Technologies, Inc. 1 Introduction Diffusion models [ 12 , 33 ] have emerged as powerful generative models capable of representing complex data distributions, driving significant advancements in high-fidelity image generation [ 27 , 30 , 25 , 40 ] . However, real-world use cases often demand personalized generation, where the model generates not only generic concepts but also specific concepts like a novel character or person [ 28 , 9 , 21 ] . This personalization technique is particularly useful for applications such as portrait editing and specific content creation. Many approaches have been proposed to personalize diffusion models for novel concepts. One line of work [ 31 , 36 , 29 , 6 ] learns a general-purpose encoder capable of adapting to new concepts without test-time finetuning. However, this approach requires large-scale text-image datasets and considerable computational resources, and must often be re-trained from scratch for each pre-trained model, making it expensive to adapt to the growing variety of text-to-image or text-to-video diffusion architectures. An alternative solution is to directly fine-tune a pre-trained diffusion model on the target images, which can be roughly categorized into fine-tuning low-rank adapters [ 14 , 5 ] , text embeddings [ 9 ] , specific layers [ 15 , 21 ] , or the full model parameters [ 28 ] . This strategy can be applied to any pre-trained diffusion model and only requires a few target images, proving effective for personalization in various scenarios. Despite these advancements, optimization-based personalization remains a nontrivial challenge. When fine-tuning a model to a specific, often much smaller, target data distribution, there is an inevitable tradeoff between aligning with the target distribution ( e.g . , learning a novel concept with a few target images) and preserving broad knowledge of the original model ( e.g . , preserving text editability). This trade-off manifests in various ways: a fine-tuned model for personalization may generate high-quality domain-specific images but lose text editability, making it difficult to generalize across different prompts. Conversely, a model that retains broad knowledge from its original training distribution may struggle to fully align with new concepts introduced in fine-tuning. Existing approaches attempt to mitigate this trade-off by adjusting the number of fine-tuning iterations [ 9 , 28 ] , leveraging regularization [ 28 , 15 ] , reducing the model parameters of updates [ 14 , 5 ] , yet they primarily operate at the fine-tuning stage. Moreover, balancing the trade-off between comprehensive text editability and precise alignment with a new target distribution remains an open problem. A parallel line of research focuses on sampling guidance techniques, such as Classifier-Free Guidance (CFG) [ 11 ] and AutoGuidance (AG) [ 19 ] . These methods rely on “weak models” to detect and push the outputs away from poor trajectories, thereby guiding samples toward higher-quality regions of the data manifold. For instance, CFG uses an unconditional model using the null text prompt as the weak model, while AG uses an unlearned model—one with fewer parameters or less training—conditioned on the same text prompt. However, these methods have limitations when applied to personalized diffusion models, as shown in Fig. 1 . CFG can interfere with the model’s adaptation to the target distribution, restricting its ability to fully utilize fine-tuned knowledge. Moreover, while the guidance scale of CFG adjusts the degree of text fidelity, it does not improve alignment with the target data. On the other hand, AG, which primarily enhances alignment with the target distribution, however, lacks an explicit mechanism for improving text alignment, making it unsuitable for fine-tuned text-to-image models. Figure 1 : Motivation of Personalization Guidance. Classifier-free guidance [ 11 ] can interfere with subject fidelity due to its alignment with the text prompt. In contrast, autoguidance [ 19 ] fails to reflect the text prompt but generates the subject more similarly. Our guidance complements the limitations of both methods. To address these issues, we propose a simple yet effective guidance technique, called Personalization Guidance , for personalized diffusion models, which can dynamically balance text alignment and adaptation to the target distribution. Our method leverages an “unlearned” weak model with a null text prompt, without incurring additional computational overhead during inference. Here, we aim to balance the alignment with the target distribution ( e.g . , subject fidelity) and the preservation of the broad knowledge of the original model ( e.g . , text fidelity) by controlling the extent of “unlearning” in a weak model. Here, the degree of unlearning refers to the extent to which the target data distribution has not been learned compared to the fine-tuned model. The rationale behind adjusting this degree of unlearning is to steer the output toward an optimal, well-balanced latent space. Specifically, during inference, we interpolate the weight parameters between the pre-trained and fine-tuned diffusion models to derive an optimal weak model, effectively controlling the extent of unlearning. Unlike guidance scales, which adjust the intensity of guidance, weight interpolation for the weak model serves to steer outputs in an appropriate direction. This straightforward approach can be readily integrated into various fine-tuning diffusion models. 2 Related Work Personalization of Diffusion Models. Personalizing diffusion models aim to generate novel images by adapting to specific concepts using a set of reference images. Optimization-based methods personalize models by fine-tuning textual embeddings [ 9 , 21 , 15 ] , entire model parameters with regularization [ 28 ] , or low-rank adapters [ 14 , 5 ] . Another line of research leverages extensive text-image datasets to train dedicated text-to-image personalization models, by fine-tuning additional modules [ 36 , 31 , 29 , 6 ] . Recently, several studies [ 23 , 4 , 32 ] have achieved better personalization without additional fine-tuning or training. Orthogonal to these existing approaches, we introduce a simple yet effective personalization guidance technique that seamlessly integrates with various personalization methods by modifying the classifier-free guidance. Moreover, our guidance enables