Towards More Accurate Personalized Image Generation: Addressing Overfitting and Evaluation Bias

Mingxiao Li, Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens · 2025

Read Towards More Accurate Personalized Image Generation: Addressing Overfitting and Evaluation Bias by Mingxiao Li, Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06632.txt.

Introduction

Mingxiao Li, Tingyu Qu, Tinne Tuytelaars, and Marie-Francine Moens tackle two flaws in personalized image generation: models that copy their few training photos too closely, and scorekeeping that flatters them. Their paper appeared in two thousand twenty-five. Personalization teaches a generator one subject from a handful of images. The aim is new scenes with the same subject, not replays.

Their training pipeline adds an attractor that filters distractions out of the training shots. The model can then focus on the subject itself. For fair grading, they built a fresh test set with varied, well-designed prompts. Past practice graded models on their own training images, which inflates scores. Human grading, the usual fallback, wobbles with bias. The new bench lets automatic scores be trusted again.

Catalog record

CREATORMingxiao Li, Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06632

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards More Accurate Personalized Image Generation: Addressing Overfitting and Evaluation Bias Mingxiao Li, Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens Abstract Personalized image generation via text prompts has great potential to improve daily life and professional work by facilitating the creation of customized visual content. The aim of image personalization is to create images based on a user-provided subject while maintaining both consistency of the subject and flexibility to accommodate various textual descriptions of that subject. However, current methods face challenges in ensuring fidelity to the text prompt while not overfitting to the training data. In this work, we introduce a novel training pipeline that incorporates an attractor to filter out distractions in training images, allowing the model to focus on learning an effective representation of the personalized subject. Moreover, current evaluation methods struggle due to the lack of a dedicated test set. The evaluation set-up typically relies on the training data of the personalization task to compute text-image and image-image similarity scores, which, while useful, tend to overestimate performance. Although human evaluations are commonly used as an alternative, they often suffer from bias and inconsistency. To address these issues, we curate a diverse and high-quality test set with well-designed prompts. With this new benchmark, automatic evaluation metrics can reliably assess model performance. 1 1 1 Code and dataset are avaiable at https://github.com/Mingxiao-Li/Towards-More-Accurate-Personalized-Image-Generation 1 Introduction Personalized image generation [ 9 , 34 , 1 ] aims to create images that incorporate a user-specified subject or concept in a set of images. The generation adheres to the provided text prompts, while consistently generating the same subject and its appearance. This technology transforms creative and professional workflows by generating customized visuals from text prompts. Recent advances in text-to-image models enhance quality and adaptability, enabling more accurate, coherent results. However, despite these advancements, challenges remain. Figure 1 : Changes in DINO image-image similarity and CLIP text-image similarity scores for the DreamBooth model across training steps. The DINO score, evaluated on a separate test set, shows a stronger correlation with the CLIP text similarity score. In contrast, the DINO score on the training set remains nearly unchanged, even after the model has fully overfit to the training data. One major issue is ensuring fidelity to the text prompt while not overfitting to the training data. Models that overfit may generate images highly similar to those in the training set rather than truly understanding and generalizing the provided subject. Furthermore, the current automatic evaluation framework for personalized image generation is flawed. Automatic metrics such as CLIP text-image similarity score, CLIP image-image similarity score, and DINO image-image score [ 30 , 4 , 34 ] are widely used to assess both text alignment and subject consistency. However, due to the absence of an independent test set, in previous works these metrics are always computed using the training data itself [ 9 , 34 , 1 ] . This leads to evaluation biases. Models that overfit the training set can achieve artificially high similarity scores. For example, in Fig. 1 , the higher similarity score of the generated image at 5000 steps misleadingly suggests strong generalization. Addressing such evaluation flaws is crucial for accurately tracking progress in personalized image generation and ensuring that models learn beyond memorizing training data. The goal of image personalization is to generate images based on a user-provided subject while ensuring both subject consistency and adaptability to diverse textual descriptions that regard that same subject. To tackle the aforementioned challenges, we propose a novel training pipeline that seamlessly integrates with existing tuning-based image personalization techniques, such as Textual Inversion [ 9 ] and NeTI [ 1 ] . Unlike traditional approaches that solely focus on learning the target subject, our pipeline introduces an attractor, a mechanism specifically designed to filter out distractions in training images. By reducing background noise and emphasizing the primary subject, this method enables more effective subject learning, resulting in higher-quality image generation and improved alignment with user prompts. Additionally, our approach mitigates the risk of overfitting while preserving the model’s ability to generate diverse and highly personalized content. To address the evaluation bias caused by using training data for automatic metric computation, we curate a diverse and high-quality dataset consisting of both a dedicated training set and a separate test set with carefully designed, detailed prompts. This benchmark enables reliable performance assessment by ensuring that evaluation metrics are computed on unseen data. This reduces the risk of overestimated results and mitigates flaws of existing evaluation practices. In this paper, we present our novel training framework alongside our newly developed benchmark for personalized image generation. We demonstrate that our approach improves subject fidelity while reducing overfitting and enhances the reliability of automatic evaluation metrics. Our contributions can be summarized as follows: • A novel training pipeline that utilizes an attractor to filter distractions, improving the focus on primary subjects in personalized image generation. • A curated benchmark dataset with diverse and high-quality prompts, enabling more reliable automatic evaluation of model performance. • Comprehensive analysis of existing evaluation methods, highlighting their limitations and demonstrating how our benchmark mitigates these issues. 2 Related Work 2.1 Diffusion Models Diffusion models are probabilistic generative models that have achieved remarkable success in learning complex data distributions across various domains, including images [ 33 , 31 , 6 , 27 ] , videos [ 50 , 28 , 54 , 43 , 5 , 51 , 14 , 3 ] , and 3D objects [ 29 , 19 , 45 , 49 , 39 ] . These models operate through a two-step process: a forward process, which incrementally adds noise to a clean image, and a backward process, which progressively removes the noise to reconstruct the original image. This iterative generative mechanism not only enables high-quality synthesis but also offers significant flexibility for controllable generation, such as image layout control [ 52 , 22 ] and video motion control [ 18 , 10 ] . Beyond creative applications, diffusion models have also found widespread use in scientific domains, including protein generation [ 8 , 40 , 47 , 2 ] and brain visual decoding [ 37 , 36 , 7 , 38 ] . However, despite their strong generative capabilities, diffusion models suffer from slow sampling speeds, making them computationally expensive. To address this, researchers have explored methods to accelerate the generative process [ 35 , 21 , 53 ] . Meanwhile, another line of research focuses on improving generation quality by mitigating exposure bias [ 17 , 23 ] . 2.2 Personalization To achieve image personalization, existing approaches can be broadly categorized into fine-tuning-based methods and encoder-based methods. Early works, including [ 9 , 34 , 1 , 11 , 42 , 15 ] , primarily focus on fine-tuning techniques. These methods adjust either specific parts or the entire set of weights in the diffusion network while leveraging pseudo-word embeddings to store and represent the target subject effectively. In contrast, encoder-based methods [ 12 , 16 , 24 , 25 , 46 ] eliminate the need for test-time tuning by introducing an additional network—often referred to as an adapter or subject encoder—that learns to encode subject information. While this approach enables faster inference, it requires training

Read the complete text