Parallel Rescaling: Rebalancing Consistency Guidance for Personalized Diffusion Models
JungWoo Chae, Jiyoon Kim, Sangheum Hwang · 2025
Read Parallel Rescaling: Rebalancing Consistency Guidance for Personalized Diffusion Models by JungWoo Chae, Jiyoon Kim, Sangheum Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00607.txt.
Introduction
JungWoo Chae, Jiyoon Kim, and Sangheum Hwang wrote this 2025 arXiv paper on personalizing image generators without wrecking their obedience to prompts. Tools that teach a model a specific person or object from a few photos tend to overfit: the face survives, but the requested scene warps. A recent fix steers generation for consistency, yet still falters on complex or stylized prompts.
The team's method, parallel rescaling, splits that steering signal into two parts: one aligned with the model's ordinary prompt guidance, and one crossing it. Only the aligned part is turned down, which cuts the interference while keeping the subject's identity. No extra training data or costly labels are needed. Across tests, including tricky stylized prompts, the balance of likeness and prompt fidelity beat earlier methods. The work targets anyone generating images of a specific subject on demand.
Catalog record
| CREATOR | JungWoo Chae, Jiyoon Kim, Sangheum Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00607
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Parallel Rescaling: Rebalancing Consistency Guidance for Personalized Diffusion Models JungWoo Chae, Jiyoon Kim, Sangheum Hwang Abstract Personalizing diffusion models to specific users or concepts remains challenging, particularly when only a few reference images are available. Existing methods such as DreamBooth and Textual Inversion often overfit to limited data, causing misalignment between generated images and text prompts when attempting to balance identity fidelity with prompt adherence. While Direct Consistency Optimization (DCO) with its consistency-guided sampling partially alleviates this issue, it still struggles with complex or stylized prompts. In this paper, we propose a parallel rescaling technique for personalized diffusion models. Our approach explicitly decomposes the consistency guidance signal into parallel and orthogonal components relative to classifier-free guidance (CFG). By rescaling the parallel component, we minimize disruptive interference with CFG while preserving the subject’s identity. Unlike prior personalization methods, our technique does not require additional training data or expensive annotations. Extensive experiments show improved prompt alignment and visual fidelity compared to baseline methods, even on challenging stylized prompts. These findings highlight the potential of parallel rescaled guidance to yield more stable and accurate personalization for diverse user inputs. 1 Introduction Text-to-image diffusion models Podell et al. 2024 ; Esser et al. 2024 have transformed content creation by enabling users to generate vivid, imaginative visuals simply from textual prompts. Recently, personalization techniques such as DreamBooth Ruiz et al. 2023 and Textual Inversion Gal et al. 2022 have expanded these capabilities further, allowing a user to incorporate a custom subject—e.g., a specific person, animal, or product—into generated images after fine-tuning with only a few examples. However, these personalized models often suffer from text misalignment : they overfit to the limited training images, sometimes ignoring or overriding aspects of the prompt and unintentionally recreating backgrounds or styles from the reference set. A notable effort to address overfitting is Direct Consistency Optimization (DCO) Lee et al. 2024 , which introduces a consistency function to constrain the fine-tuned diffusion model so that its outputs remain close to those of the base (pre-trained) generator. Although DCO effectively balances identity fidelity with prompt adherence , it can still fail to perform reliably on long or complex prompts , as well as highly stylized descriptions. Meanwhile, other approaches typically require extensive resources, such as costly segmentation masks Avrahami et al. 2023 ; Jang et al. 2024 or high-VRAM hardware Pang et al. 2024 ; Zhang et al. 2024 , making them less accessible to the average user. In this work, we propose a simple yet effective method for personalization that preserves subject identity and offers stronger alignment with complex prompts. Our key insight is to analyze the consistency guidance signal by decomposing it into parallel and orthogonal components with respect to classifier-free guidance (CFG) Ho and Salimans 2021 . Through this decomposition, we discover that the parallel component, which is essential for maintaining the target subject’s features, can inadvertently conflict with text guidance—especially during denoising steps where stylization plays a critical role. Based on these observations, we introduce a parallel rescaling strategy that mitigates interference in CFG-based guidance, thereby improving prompt fidelity while retaining subject identity. Our contributions are: • We perform a detailed analysis of the consistency guidance signal, showing how decomposing it into parallel and orthogonal components reveals the source of text misalignment in personalized diffusion. • We propose a straightforward parallel rescaling technique that re-centers and re-scales the parallel component of consistency guidance. This ensures stronger alignment with complex prompts while retaining subject features. Through experiments, we demonstrate that our approach reliably preserves the subject’s identity while improving text alignment, even under elaborate or artistic prompts. By requiring only minimal computational overhead, our parallel rescaling guidance represents a promising step toward more accessible and robust personalized diffusion models. 2 Related Work Diffusion Models and Personalization. Recent diffusion models Podell et al. 2024 ; Betker et al. 2023 ; Esser et al. 2024 have proven highly effective for text-to-image generation, creating diverse and high-fidelity images. Although these models exhibit impressive creative range, direct application to personalized content remains challenging. Two leading approaches for personalization are DreamBooth Ruiz et al. 2023 and Textual Inversion Gal et al. 2022 . Building on these methods, recent work Kumari et al. 2023 ; Nam et al. 2024 ; Voynov et al. 2023 ; Arar et al. 2024 ; Alaluf et al. 2023 ; Qiu et al. 2023 has further expanded the field with techniques such as subject-driven generation Wei et al. 2023 , identity-preserving diffusion Ma et al. 2023 , and segmentation-based personalization Avrahami et al. 2023 ; Jang et al. 2024 . Both traditional and newer techniques highlight an ongoing tension between identity fidelity and prompt fidelity in personalized diffusion. Guidance Methods. To steer text-to-image diffusion models during sampling, Classifier-Free Guidance (CFG) Ho and Salimans 2021 scales the difference between conditional and unconditional predictions, boosting alignment with the user’s prompt. While CFG often improves adherence, excessive scaling can distort visual quality and reduce diversity. Various enhancements address these limitations: for instance, autoguidance Karras et al. 2024 adaptively tunes the guidance scale to prevent mode collapse, guidance interval Kynkäänniemi et al. 2024 selectively applies CFG at specific timesteps for smoother sampling. More recent approaches like Attend-and-Excite Chefer et al. 2023 further refine guidance by leveraging attention mechanisms. However, these strategies primarily target generic text-to-image tasks and do not inherently resolve the personalization trade-off between subject preservation and stylistic or prompt-based variation. Direct Consistency Optimization (DCO) Lee et al. 2024 specifically tackles this trade-off by learning a consistency function that anchors a fine-tuned model’s outputs to those of a base diffusion model. During inference, a Consistency Guidance term is added to CFG to preserve the subject’s core features. However, a fixed consistency weight can still interfere with the prompt—particularly under complex or stylized conditions. Our work builds on DCO’s insights by decomposing the consistency guidance into parallel and orthogonal components relative to CFG. Through a parallel rescaling strategy , we mitigate disruptive interactions in the parallel term, enabling finer control of prompt fidelity with minimal sacrifice of identity preservation. In doing so, we contribute to the ongoing effort to balance personalization with the versatility needed to handle complex prompts. 3 Preliminary: Consistency Guidance Classifier-Free Guidance (CFG). In diffusion models, CFG steers the generation toward a given text prompt without an external classifier. At each denoising step t , the model is run twice: once with conditional prompt c and once without prompt (the “unconditional” case). Let \epsilon_{\phi}(x_{t}|c) denote the noise prediction conditioned on c using the pretrained model (with parameters \phi ) and \epsilon_{\phi}(x_{t}|\varnothing) denote the prediction for an empty prompt. The text guidance vector is defined as: g_{\text{text}}(x_{t})=\epsilon_{\phi}(x_{t}|c)-\epsilon_{\phi}(x_{t}|\varnothing), (1) and is scaled by a factor \omega_{\text{text}}>1 (the guidance scale) to form the guided prediction: \epsilon_{\text{CFG}}(x_{t})=\epsilon_{\phi}(x_{t}|\varnothing)+\omega_{\text{text}}\,g_{\text{text}}(x_{t}). (2) Consistency Guidance in DCO. To enforce consistency with reference images, Consistency Guidance Sampling extends CFG by incorporating an additional guidance term. After fine-tuning the model to learn a consistency function that measures similarity between generated images and reference images, DCO defines a consistency condition c_{\text{cons}} that anchors the model to the base concept. The consistency guidance vector is given by: g_{\text{cons}}(x_{t})=\epsilon_{\theta}(x_{t}\,|\,c)-\epsilon_{\phi}(x_{t}|\,c), (3) where \epsilon_{\theta} represents the noise prediction from the personalized (fine-tuned) model. The final sampler