ACCORD: Alleviating Concept Coupling through Dependence Regularization for Text-to-Image Diffusion Personalization

Shizhan LiuAnt GroupHao ZhengAnt GroupHang YuAntGroup, Jianguo LiAntGroup · 2025

Read ACCORD: Alleviating Concept Coupling through Dependence Regularization for Text-to-Image Diffusion Personalization by Shizhan LiuAnt GroupHao ZhengAnt GroupHang YuAntGroup, Jianguo LiAntGroup (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01122.txt.

Introduction

Shizhan Liu, Hao Zheng, Hang Yu, and Jianguo Li fix a flaw in image generators tuned to a user's own photos, in work from 2025. With only a few sample images, the model tangles the target concept with things that happen to sit beside it. A pet may inherit the park behind it, say.

Older fixes attack the tangle from the side, trading text control against likeness. A statistics study here traces the tangle to two distinct dependence gaps. Two add-on losses each close one gap, one during denoising and one on the prior. Tests across settings show a better balance of prompt obedience and personal likeness. The losses plug into existing pipelines.

Catalog record

CREATORShizhan LiuAnt GroupHao ZhengAnt GroupHang YuAntGroup, Jianguo LiAntGroup
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01122

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ACCORD: Alleviating Concept Coupling through Dependence Regularization for Text-to-Image Diffusion Personalization Shizhan LiuAnt GroupHao ZhengAnt GroupHang YuAntGroup, Jianguo LiAntGroup Abstract Image personalization has garnered attention for its ability to customize Text-to-Image generation using only a few reference images. However, a key challenge in image personalization is the issue of conceptual coupling, where the limited number of reference images leads the model to form unwanted associations between the personalization target and other concepts. Current methods attempt to tackle this issue indirectly, leading to a suboptimal balance between text control and personalization fidelity. In this paper, we take a direct approach to the concept coupling problem through statistical analysis, revealing that it stems from two distinct sources of dependence discrepancies. We therefore propose two complementary plug-and-play loss functions: Denoising Decouple Loss and Prior Decouple loss, each designed to minimize one type of dependence discrepancy. Extensive experiments demonstrate that our approach achieves a superior trade-off between text control and personalization fidelity. 1 Introduction The advancement of Text-to-Image (T2I) Diffusion Models [ 1 , 2 ] has lowered the barrier to generating high-quality and imaginative images from text prompts. However, pretrained T2I models often struggle to accurately produce personalized images, such as those depicting private pets or unique artistic styles. As a result, personalized image generation has recently gained significant attention, necessitating users to provide several reference images pertaining to the personalization target, which enables T2I models to create images of the personalization target based on text prompts. The primary challenge of image personalization is “concept coupling”. Due to the limited availability of reference images for the personaliztion target, the model tends to confuse the target with other concepts that appear alongside it in these images. This entanglement hinders the model’s ability to accurately control the attributes associated with the personalization target based on text. For example, as shown in Fig. 1 , the model may interpret "a person carrying a backpack" as the primary focus, rather than "backpack," because these elements frequently co-occur in the reference images. Consequently, the generated images often deviate from the intended text prompts, frequently including an unintended person in the output. Figure 1: Illustration of the concept coupling problem. The personalization target is a “backpack”, but in the reference images, the backpack and a girl always appear together. This causes the model finetuned without concept decoupling to frequently generate an additional girl and not fully adhere to the prompt. Statistically, the co-occurrence of “backpack” and “girl” in generated images is significantly higher than the inherent concept dependence. However, existing image personalization methods fail to directly address the issue of concept coupling. The current methods for mitigating concept coupling can be broadly categorized into three types: data regularization, weight regularization, and loss regularization. Data regularization methods [ 3 , 4 ] , generate or retrieve a regularization dataset based on the superclass of the personalization target to preserve the model’s prior knowledge during training. Nevertheless, the limited size of the regularization dataset cannot ensure that inter-concept relationships align with their inherent relationships, and simultaneous training on regularization and reference images often compromises personalization fidelity. Weight regularization approaches, like SVDiff [ 5 ] and OFT [ 6 ] , mitigate the risk of overfitting to reference images by constraining the optimization space of parameters, such as by fine-tuning only the cross-attention layer parameters. Such constraints do not guarantee selective learning of the personalization target and other concepts, and the personalization fidelity may degrade as parameters are constrained. Loss regularization methods [ 7 , 8 ] introduce auxiliary loss functions for regularization, but existing methods rely on empirical choices for optimization objectives and lack theoretical guarantees for reducing concept coupling. In this paper, we directly frame concept coupling as a statistical problem, where personalized Text-to-Image (T2I) models create unintended dependencies between the personalization target and other concepts in reference images. As illustrated in Fig. 1 , when personalizing for concepts like ’backpack’ alongside ’girl’, the model learns artificially strong associations that exceed their natural relationship in the training data. We analyze this concept coupling by breaking it down into two components: the Denoising Dependence Discrepancy and the Prior Dependence Discrepancy . To minimize these discrepancies, we propose ACCORD , a plug-and-play method that employs two specialized loss functions for dependence regularization. Specifically, we propose the Denoising Decouple Loss (DDLoss) and the Prior Decouple Loss (PDLoss) to reduce the denoising dependence discrepancy and prior dependence discrepancy, respectively. This enables ACCORD to eliminate the need for regularization datasets and excessive weight regularization as in previous works; instead, it directly minimizes over-dependencies between concepts. The DDLoss functions by establishing an upper bound on the denoising dependence discrepancy, aggregating the discrepancies between consecutive denoising steps and utilizing the diffusion model as an implicit classifier to minimize these step-wise differences. Complementing this, the PDLoss addresses cases where training the personalization target’s representation [ 9 , 10 ] increases the prior dependence discrepancy, using CLIP’s [ 11 ] conditional classification capabilities to minimize this effect. Experiments demonstrate that the proposed loss functions alleviate the concept coupling issue in image personalization more effectively, achieving a better balance between text control and personalization fidelity. Our contributions can be summarized as follows: • We formally characterize concept coupling in image personalization as a statistical problem of unintended dependencies and propose ACCORD, a plug-and-play method that directly addresses concept coupling without requiring regularization datasets or extensive weight constraints. • We identify two distinct sources of dependence discrepancies in concept coupling: Denoising Dependence Discrepancy and Prior Dependence Discrepancy. To address these discrepancies, we propose Denoising Decouple Loss and Prior Decouple Loss, respectively. • Experimental results demonstrate the superiority of ACCORD in image personalization. Moreover, the proposed losses prove effective in zero-shot conditional control tasks, highlighting the potential of concept decoupling. 2 Related Works Test Time Finetuning-based Image Personalization. Test-time fine-tuning methods personalize pre-trained T2I models by adapting them to reference images of the personalization target. Although this approach requires time and computational resources for training, it achieves greater flexibility for diverse personalization demands and often strikes a better balance between text control and personalization fidelity. Therefore, this paper focuses on improving the test-time fine-tuning method. In terms of addressing the concept coupling problem, test-time fine-tuning methods can be broadly categorized into three types: data regularization, weight regularization, and loss regularization. In data regularization methods, Dreambooth [ 3 ] mitigates overfitting by using a pretrained T2I model to generate a set of images for the superclass of the personalization target and trains on both reference and regularization images. Custom Diffusion [ 4 ] enhances the quality of regularization images by retrieving them from real images. Specialist Diffusion [ 12 ] designs extensive data augmentation techniques. However, the limited size of the regularization dataset hampers its ability to accurately capture the inherent relationships among concepts, misaligning the optimization objective with reducing concept coupling. Furthermore, simultaneous training on regularization and reference images often impairs personalization fidelity due to their substantial appearance differences. Weight regularization methods [ 9 , 13 , 5 , 6 ] finetune the parameters of the T2I model in a constrained manner, such as by adjusting only the text embedding of the personalization target or the singular values of the weight matrices.

Read the complete text