APT: Adaptive Personalized Training for Diffusion Models with Limited Data

JungWoo Chae 1 Jiyoon Kim 1 JaeWoong Choi 1 Kyungyul Kim 1 Sangheum Hwang 2 1 LG CNS AI Research 2 Department of Data Science, Seoul National University of Science and Technology {cjwoolgcns, jiyoonkim, jaewoong_choi, kyungyul.kim }@lgcns.com [email protected] · 2025

Read APT: Adaptive Personalized Training for Diffusion Models with Limited Data by JungWoo Chae 1 Jiyoon Kim 1 JaeWoong Choi 1 Kyungyul Kim 1 Sangheum Hwang 2 1 LG CNS AI Research 2 Department of Data Science, Seoul National University of Science and Technology {cjwoolgcns, jiyoonkim, jaewoong_choi, kyungyul.kim }@lgcns.com [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-02687.txt.

Introduction

JungWoo Chae and co-authors present a 2025 method for tailoring image models on very little data. Diffusion models can learn a new face or style from a few sample images. With so few samples, they tend to overfit: they copy the samples and lose range and text sense. The team's method, called APT, fights that drift during fine-tuning.

Three parts work as a team. A gauge flags how much overfit builds at each step, and the method answers with more sample variety and shifted loss weights. A second part holds the model's inner feature stats near their start values. A third aligns attention maps with the base model to keep prior knowledge. In wide tests, APT beat prior methods on image quality and range. Personal models could thus learn from a handful of photos without going stale.

Catalog record

CREATORJungWoo Chae 1 Jiyoon Kim 1 JaeWoong Choi 1 Kyungyul Kim 1 Sangheum Hwang 2 1 LG CNS AI Research 2 Department of Data Science, Seoul National University of Science and Technology {cjwoolgcns, jiyoonkim, jaewoong_choi, kyungyul.kim }@lgcns.com [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-02687

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

APT: Adaptive Personalized Training for Diffusion Models with Limited Data JungWoo Chae 1 Jiyoon Kim 1 JaeWoong Choi 1 Kyungyul Kim 1 Sangheum Hwang 2 1 LG CNS AI Research 2 Department of Data Science, Seoul National University of Science and Technology {cjwoolgcns, jiyoonkim, jaewoong_choi, kyungyul.kim }@lgcns.com [email protected] Abstract Personalizing diffusion models using limited data presents significant challenges, including overfitting, loss of prior knowledge, and degradation of text alignment. Overfitting leads to shifts in the noise prediction distribution, disrupting the denoising trajectory and causing the model to lose semantic coherence. In this paper, we propose Adaptive Personalized Training (APT) , a novel framework that mitigates overfitting by employing adaptive training strategies and regularizing the model’s internal representations during fine-tuning. APT consists of three key components: (1) Adaptive Training Adjustment , which introduces an overfitting indicator to detect the degree of overfitting at each time step bin and applies adaptive data augmentation and adaptive loss weighting based on this indicator; (2) Representation Stabilization , which regularizes the mean and variance of intermediate feature maps to prevent excessive shifts in noise prediction; and (3) Attention Alignment for Prior Knowledge Preservation , which aligns the cross-attention maps of the fine-tuned model with those of the pretrained model to maintain prior knowledge and semantic coherence. Through extensive experiments, we demonstrate that APT effectively mitigates overfitting, preserves prior knowledge, and outperforms existing methods in generating high-quality, diverse images with limited reference data. 1 Introduction The advent of diffusion models has significantly advanced the field of generative modeling, enabling the synthesis of diverse and high-quality images Ho et al. 2020 ; Song et al. 2020 ; Rombach et al. 2022 . Personalization techniques, such as DreamBooth Ruiz et al. 2023 and Textual Inversion Gal et al. 2022 , have further enhanced these models by enabling subject-driven generation tailored to specific user needs. Such advancements have broad applications, from artistic content creation to specialized data augmentation in machine learning tasks Zeng et al. 2024 ; Zhang et al. 2024 . However, personalizing diffusion models using limited data presents significant challenges. One critical issue is overfitting, which causes excessive shifts in the noise prediction distribution, disrupting the denoising trajectory of the pretrained diffusion model Ruiz et al. 2023 ; Lee et al. 2024 . These shifts lead to the loss of prior knowledge, degradation of text alignment, and a reduced ability of the model to generalize to unseen prompts. Overfitting may cause the model to memorize spatial layouts, resulting in generated images with overly similar compositions, or to over-memorize textures, leading to poor stylization and a lack of diversity in response to different prompts. Moreover, overfitting can lead to the loss of prior knowledge, causing the model to generate images that do not accurately reflect the desired concept or context. For example, when generating an image with “a photo of a backpack” , pretrained diffusion models may naturally include a person carrying the backpack, leveraging prior knowledge about common contexts. However, after fine-tuning with limited data, the model may lose this prior knowledge, resulting in images of only the backpack without a person. This loss of prior knowledge is accompanied by changes in the cross-attention maps, which further degrade the quality and coherence of the generated images. Existing methods Lee et al. 2024 ; Shiohara and Yamasaki 2024 ; Han et al. 2023 ; Zhang et al. 2024 ; Pang et al. 2024 have addressed these challenges through various regularization techniques and novel fine-tuning approaches. Techniques that constrain attention using masks Avrahami et al. 2023 often require additional annotations and may not align effectively with the soft attention distributions of the model. Furthermore, prior preservation techniques that incorporate additional data Ruiz et al. 2023 fine-tune the model by combining subject and auxiliary images but often suffer from overfitting. This can disrupt the original denoising trajectory, resulting in overfitting to auxiliary datasets and reduced generalization in generation quality Lee et al. 2024 . In this work, we propose Adaptive Personalized Training (APT) , a novel framework that addresses these challenges by mitigating overfitting with adaptive training strategies, regularizing the internal representations of the model during fine-tuning, and preserving prior knowledge. Specifically, our method consists of three key components: 1. Adaptive Training Adjustment : We introduce an overfitting indicator to detect the degree of overfitting and apply adaptive data augmentation and loss weighting based on this indicator. This approach addresses the varying influence of diffusion model parameters across different time steps due to the beta scheduling, effectively mitigating overfitting and adjusting the training dynamics. 2. Representation Stabilization : We regularize the significant shifts in the noise prediction \boldsymbol{\epsilon} caused by overfitting by constraining the mean and variance of the intermediate feature maps. This helps preserve the statistical properties of the representations of the pretrained model. 3. Attention Alignment for Prior Knowledge Preservation : To maintain the prior knowledge in the text embeddings, we propose regularizing the cross-attention maps. By aligning the attention distributions of the fine-tuned model with those of the pretrained model, we ensure that the model retains semantic coherence. Our contributions can be summarized as follows: • We introduce Adaptive Personalized Training (APT) , a novel method that addresses overfitting and the loss of prior knowledge in diffusion model personalization with limited data. APT incorporates adaptive training adjustments, representation stabilization, and attention alignment to mitigate overfitting and preserve prior knowledge during fine-tuning. • Through extensive experiments, we demonstrate that APT outperforms existing methods in preserving the text alignment ability and prior knowledge of pretrained models, while generating high-quality and diverse personalized images. Our method provides a cohesive solution that addresses both the varying influence of model parameters across time steps and the internal representation shifts that arise during fine-tuning with limited data. By mitigating overfitting and preserving prior knowledge, we enable the model to generalize better to unseen prompts while accurately capturing the desired concepts from the reference data. 2 Related Work Text-to-Image Personalization Recent advances in diffusion models have enabled high-quality image synthesis through large-scale datasets and advanced architectures Rombach et al. 2022 ; Saharia et al. 2022b ; Ramesh et al. 2022 ; Betker et al. 2023 ; Yu et al. 2022 ; Nichol et al. 2021 ; Saharia et al. 2022a , with techniques like classifier-free guidance Ho and Salimans 2021 ; Chung et al. 2024b ; Shen et al. 2024 enhancing text alignment. Personalization of text-to-image models adapts pretrained models to represent new concepts based on user-provided images. Key methods include DreamBooth Ruiz et al. 2023 , which fine-tunes the entire model for high fidelity, and Textual Inversion Gal et al. 2022 , which optimizes textual embeddings without altering model weights for efficiency. Parameter-efficient fine-tuning methods, such as LoRA Hu et al. 2022 , Custom Diffision Kumari et al. 2023 , and Svdiff Han et al. 2023 , update only a small subset of parameters to reduce resource demands while maintaining quality. Recent advances like P+ Voynov et al. 2023 and NeTI Alaluf et al. 2023 expand textual conditioning spaces, enabling greater control and expressiveness without full model fine-tuning, achieving faster convergence and improved editability. Regularization in T2I Personalization Maintaining the prior knowledge of pretrained models during personalization is essential to prevent concept drift.

Read the complete text