ConceptGuard : Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation
Zirun Guo, Tao Jin · 2025
Read ConceptGuard : Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation by Zirun Guo, Tao Jin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10358.txt.
Introduction
Zirun Guo and Tao Jin present a 2025 paper on teaching image generators one personal concept after another without wrecking what came before. Tools that learn a pet or a face from a few photos usually train on all concepts at once. In real use, concepts arrive in sequence, and later training can erase or blur earlier ones.
Their system, ConceptGuard, fights both failures with three tools: shifted embeddings, prompts that bind each concept to its own token, and a penalty that guards old knowledge. A priority queue tracks how important and how recent each concept is. Across wide tests, the authors report clear wins over all rival methods on both numbers and sample images.
Catalog record
| CREATOR | Zirun Guo, Tao Jin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10358
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ConceptGuard : Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation Zirun Guo, Tao Jin Abstract Diffusion customization methods have achieved impressive results with only a minimal number of user-provided images. However, existing approaches customize concepts collectively, whereas real-world applications often require sequential concept integration. This sequential nature can lead to catastrophic forgetting, where previously learned concepts are lost. In this paper, we investigate concept forgetting and concept confusion in the continual customization. To tackle these challenges, we present ConceptGuard , a comprehensive approach that combines shift embedding, concept-binding prompts and memory preservation regularization, supplemented by a priority queue which can adaptively update the importance and occurrence order of different concepts. These strategies can dynamically update, unbind and learn the relationship of the previous concepts, thus alleviating concept forgetting and confusion. Through comprehensive experiments, we show that our approach outperforms all the baseline methods consistently and significantly in both quantitative and qualitative analyses. 1 Introduction Text-to-Image (T2I) diffusion models ( Nichol et al. 2022 ; Rombach et al. 2022 ; Saharia et al. 2022 ; Podell et al. ) have emerged as a promising approach in the field of generative artificial intelligence, enabling the creation of high-quality images from textual descriptions. These models find a wide range of applications across various domains. Among their many uses, customization stands out as one of the most practical and popular applications. By harnessing the capabilities of T2I diffusion models, diffusion customization allows for the generation of user-defined concepts, achieving impressive results ( Ruiz et al. 2023 ; Gal et al. ; Kumari et al. 2023 ; Li et al. 2024 ; Wei et al. 2023 ; Lin et al. ) . However, in real-world applications, the new concepts we aim to customize typically arrive in a sequence. Existing methods ( Ruiz et al. 2023 ; Gal et al. ; Kumari et al. 2023 ; Wei et al. 2023 ; Lin et al. ) primarily focus on how to personalize these concepts collectively. This will give rise to problems when they are applied to a dynamic and continual environment. For example, as shown in Figure 1 , when generating previous concepts, these methods might experience catastrophic forgetting ( McCloskey and Cohen 1989 ) or concept confusion. Concretely, catastrophic forgetting occurs when the model fails to retain previously learned concepts, resulting in an inability to generate them accurately. Concept confusion arises when the model struggles to differentiate between earlier concepts, leading to outputs that blend elements of different concepts. In such continual settings, when fine-tuned on new concepts, the update of model parameters will influence the generation of previous concepts, thus leading to these issues. Figure 1 : Examples of catastrophic forgetting and concept confusion in existing methods under the continual training setting. To address the forgetting problem in continual diffusion customization, Continual Diffusion ( Smith et al. 2023 ) proposes C-LoRA, composed of a continually self-regularized low-rank adaptation in cross attention layers of the diffusion models. However, Continual Diffusion only explores how to deal with the forgetting problem, overlooking the concept confusion in the continual environment. As shown in Figure 5 , combinations of multiple concepts are still generated wrongly or blended together. Based on these observations, we propose ConceptGuard , consisting of several simple yet effective strategies to address these issues. Specifically, we propose shift embeddings, concept-binding prompts and memory preservation regularization along with a priority queue which connects all the strategies. Shift embedding helps to dynamically update the concept embedding in the continual environment. Concept-binding prompt strategy consists of concept focus, chrono-concept composition and concept-binding prompts, which helps to unbind the concepts and assess the importance and relationships of different concepts. Memory preservation regularization prevents the model from updating too fast which leads to catastrophic forgetting. Furthermore, we employ a priority queue capable of adaptively assessing the importance and occurrence order of different concepts to facilitate concept replay. We conduct extensive experiments to validate the effectiveness of our method. Experimental results indicate that in a continual environment, whether for single-concept image generation or multi-concept image generation, our proposed method consistently outperforms existing approaches in both quantitative and qualitative analyses, producing images of high quality. Ablation experiments are then conducted to validate the effectiveness of different strategies of our method. Our contributions can be summarized as: • In addition to catastrophic forgetting, we observe that concept confusion in continual customization causes elements to blend, a limitation that existing methods have yet to overcome. • To mitigate concept forgetting and confusion, we propose a comprehensive approach that combines shift embedding, concept-binding prompts, and memory preservation regularization, supplemented by a priority queue. • Our method consistently outperforms existing approaches in both quantitative and qualitative analyses, producing images of high quality. 2 Related Work Text-to-Image Diffusion Models. Diffusion models ( Nichol et al. 2022 ; Rombach et al. 2022 ; Podell et al. ) have demonstrated remarkable abilities in generation tasks. Text-to-Image (T2I) diffusion models aims to generate images from textual descriptions generated by a pre-trained text encoder. GLIDE ( Nichol et al. 2022 ) adopts CLIP guidance and classifier-free guidance for text-conditioned image synthesis. Imagen ( Saharia et al. 2022 ) leverages pre-trained large language models to provide rich textual information for image generation. Latent diffusion models such as Stable Diffusion ( Rombach et al. 2022 ) propose to project images into latent representations and add text as conditional information to the denoising process. Stable Diffusion XL ( Podell et al. ) further expands the parameters of the model and achieves better results. Customization in Diffusion Models. Diffusion customization ( Ruiz et al. 2023 ; Gal et al. ; Kumari et al. 2023 ; Li et al. 2024 ; Wei et al. 2023 ; Lin et al. ; Lin et al. 2024 ) aims to generate user-defined concepts under various contexts. Textual Inversion ( Gal et al. ) adds tokens of new concepts to dictionary and fine-tunes these new tokens. DreamBooth ( Ruiz et al. 2023 ) proposes a class-specific prior preservation loss to safeguard prior knowledge and conducts fine-tuning of all parameters of Stable Diffusion. Custom Diffusion ( Kumari et al. 2023 ) proposes to fine-tune the cross attention layers and combine multiple fine-tuned models into one via closed-form constrained optimization. However, in real-world continual applications where new concepts come continually, fine-tuning the model to new concepts would give rise to forgetting of previous customized concepts. Continual Learning. Continual Learning methods can be roughly divided into three categories. (1) Regularization methods ( Kirkpatrick et al. 2017 ; Li and Hoiem 2017 ; Zenke et al. 2017 ) address catastrophic forgetting by imposing a regularization constraint to important parameters. (2) Replay-based methods ( Rebuffi et al. 2017 ; Chaudhry et al. ; Bang et al. 2021 ) store some representative samples of previous tasks in a memory buffer and retrain these samples. (3) Architecture approaches ( Yoon et al. 2018 ; von Oswald et al. 2020 ) dynamically expand the network to mitigate forgetting for different tasks. Recently, prompt learning ( Wang et al. 2022 ; Yan et al. 2024 ; Guo et al. 2025 ) emerges as a kind of efficient methods to address catastrophic forgetting. Continual Customization. Continual customization ( Smith et al.