Per-Query Visual Concept Learning
Ori Malca, Dvir Samuel, Gal Chechik · 2025
Read Per-Query Visual Concept Learning by Ori Malca, Dvir Samuel, Gal Chechik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09045.txt.
Introduction
Ori Malca, Dvir Samuel, and Gal Chechik improve text-to-image personalization in a two thousand twenty-five paper. Personalization teaches a pretrained image model a new concept, like a specific pet or product. Uses range from entertainment to personalized design and advertising.
Existing methods personalize once per concept, before any prompt is written. The authors add a second step tuned to each query and noise seed. Two loss terms, based on self-attention and cross-attention, protect the concept's identity. They build on features previously designed to capture identity in generated faces. Tests layered the step onto six personalization methods and several base models. Improvements were significant, even over earlier per-query approaches.
Catalog record
| CREATOR | Ori Malca, Dvir Samuel, Gal Chechik |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-09045
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Per-Query Visual Concept Learning Ori Malca, Dvir Samuel, Gal Chechik Abstract Visual concept learning, also known as Text-to-image personalization, is the process of teaching new concepts to a pretrained model. This has numerous applications from product placement to entertainment and personalized design. Here we show that many existing methods can be substantially augmented by adding a personalization step that is (1) specific to the prompt and noise seed, and (2) using two loss terms based on the self- and cross- attention, capturing the identity of the personalized concept. Specifically, we leverage PDM features - previously designed to capture identity - and show how they can be used to improve personalized semantic similarity. We evaluate the benefit that our method gains on top of six different personalization methods, and several base text-to-image models (both UNet- and DiT-based). We find significant improvements even over previous per-query personalization methods. 1 Introduction Visual concept learning in generative text-to-image models is the problem of teaching a new concept to a pretrained model [ 8 , 26 , 38 ] , usually from a handful of images of the target concept. This process, also known as personalization , is typically achieved by tuning small parts of a pre-trained text-to-image model models, like the embedding layer of a token [ 8 ] , or a low-rank adaptation of the attention layers (DreamBooth LoRA [ 26 , 16 ] ). Personalization methods are evaluated in two main aspects: How well they capture the visual appearance of the target concept ( image alignment ) and how well it can be controlled to agree with new text prompts ( textual alignment ). While significant progress has been made in this area, significant room is left for further improvement, especially when the number of target concept images that is available for fine-tuning is small. Here we propose to improve generation of a new visual concept using two observations. First, that the (reconstruction) losses used for personalization were typically not designed for capturing concept identity. Second, that personalization methods typically fine-tune models using a diverse set of prompts and seeds, with the goal of using the learned concept with any seed and prompt at inference time (see [ 3 , 1 ] ). To address this, we introduce a novel loss function tailored to better preserve visual identity. Our approach is inspired by recent work [ 28 ] , which found that object appearance is largely encoded in specific layers of the self-attention mechanism. These features were originally used for instance retrieval and segmentation, and here we adapt this insight to improve the training process of existing personalization methods. To further improve text-image alignment, we show that fine-tuning with a fixed prompt and seed significantly enhances generation quality. We describe an ”add-on” fine tuning procedure that can be used on top of existing methods. For instance, once a visual concept has been learned using some method, we perform feature extraction to compute identity-sensitive features from the self-attention part, and prompt-sensitive features from the cross-attention part, and update the model using a single gradient update. We find that this single update step provides significant gains in terms of visual alignment and text alignment. The procedure is simple, fast, and easy to implement. We tested it on top of six different personalization methods and backbone text-to-image diffusion models, (both SD [ 25 ] , SDXL [ 20 ] and Flux [ 13 ] ) and found that it provides consistent gains. The paper therefore makes the following contributions: (1) An identity-preserving loss based on self and cross attention. (2) New SoTA results in personalization using a single image of the target concept. (3) It demonstrates the benefit of fine-tuning with a specific seed and prompt. Figure 2 : Method Overview. We enhance personalized text-to-image models by computing self- and cross-attention features from a single denoising step of a generated image x^{\text{gen}} and a reference image x^{\text{ref}} . Using DIFT, we match these features and define losses \mathcal{L}_{\text{SA}} , \mathcal{L}_{\text{CA}} , and \mathcal{L}_{\text{LDM}} , which are combined to update the personalization tuning parameters via gradient descent. 2 Related work Text-to-image diffusion models are trained to generate samples from a conditional data distribution by progressively denoising a variable initially sampled from a Gaussian distribution. Recent advances in text-to-image synthesis have been driven by large-scale models trained on massive web-sourced datasets [ 25 , 27 , 23 , 30 , 24 , 37 ] . Personalization in generative text-to-image models involves teaching a pre-trained model a new concept using only a few images. Early approaches tackled this problem by optimizing word embeddings [ 8 , 33 , 2 , 35 ] or fine-tuning model weights [ 26 ] . While fine-tuning typically yields better subject fidelity and photorealism—thanks to its higher expressive capacity—it tends to compromise prompt fidelity and image diversity as training progresses. Subsequent methods have aimed to mitigate this trade-off. CustomDiffusion [ 12 ] constrains updates to the text-image cross-attention layers and regularizes training using retrieved class-specific images. Perfusion [ 32 ] , inspired by ROME [ 18 ] , applies a gated rank-one update to the key and value projection matrices, anchoring the key space within the subject’s super-category to curb overfitting. DBlend [ 22 ] guides generations from an overfit checkpoint by matching the cross-attention maps of an underfit one, adjusting noise levels at each diffusion step accordingly. AttnDB [ 19 ] proposes a multi-stage training scheme that learns embedding alignment, attention maps, and subject identity separately, enhancing both identity preservation and prompt fidelity. Another line of research explores encoder-based methods [ 29 , 9 , 11 , 5 , 36 , 34 , 17 ] , which avoid subject-specific fine-tuning and model-specific weight storage. While some of these methods [ 9 , 11 , 29 , 34 ] are domain-specific (e.g., human faces or pets) and may still involve limited fine-tuning, others [ 14 , 5 ] strive for more general-purpose, zero-shot personalization. However, such models often depend on large-scale pretraining involving diffusion models in the loop. In some cases [ 5 ] , they even rely on supervision from expert-personalized models to construct their training datasets. In contrast to these encoder-based or architectural modifications, our approach remains firmly grounded in fine-tuning. We leverage the final stages of training to extract and synthesize high-quality images enriched with subject identity cues. These synthesized images are then used to train more efficient personalization models for inference-time deployment. Rather than discarding fine-tuning due to its computational cost, we embrace it as both a powerful personalization mechanism and a source of high-fidelity supervision. Our method is therefore positioned within the fine-tuning paradigm, but repurposed it to enhance prompt adherence and subject alignment while enabling efficient downstream inference—offering a unique balance between quality, adaptability, and efficiency. 3 Method 3.1 Revisiting the Loss Functions Used in Current Personalization Pipelines Most personalization methods, whether they introduce a new token in the text encoder ( textual inversion [ 8 ] ), adapt a small set of weights with low-rank updates ( DreamBooth LoRA [ 26 , 16 ] ), or fine-tune a lightweight subset of layers ( CustomDiffusion [ 12 ] ), often optimize exactly the same objective used to