Adaptive Subspace Projection for Generative Personalization

Van-Anh Nguyen, Anh Bui, Tamas Abraham, Junae Kim, Amardeep Kaur, Rollin Omari, Thuy-Trang Vu, Dinh Phung · 2026

Read Adaptive Subspace Projection for Generative Personalization by Van-Anh Nguyen, Anh Bui, Tamas Abraham, Junae Kim, Amardeep Kaur, Rollin Omari, Thuy-Trang Vu, Dinh Phung (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07257.txt.

Introduction

Van-Anh Nguyen and seven co-authors fix a failure in personalized image generation where the learned subject swallows the whole prompt. When a model learns a personal concept, it can overpower other details the user asked for. The authors trace the cause to semantic drift concentrated in a small subspace of the embedding space.

Their AdaptSP method works at test time with no training, anchoring on the stable original embedding. It isolates the drift and projects it onto the identified subspace, trimming the excess while keeping the subject's identity. Experiments show better prompt fidelity and contextual alignment than before, suggesting precise embedding adjustment beats broader fixes.

Catalog record

CREATORVan-Anh Nguyen, Anh Bui, Tamas Abraham, Junae Kim, Amardeep Kaur, Rollin Omari, Thuy-Trang Vu, Dinh Phung
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Subspace Projection for Generative Personalization Van-Anh Nguyen, Anh Bui, Tamas Abraham, Junae Kim, Amardeep Kaur, Rollin Omari, Thuy-Trang Vu, Dinh Phung Abstract Generative personalization often suffers from the semantic collapsing problem (SCP), where a learned personalized concept overpowers the rest of the text prompt, causing the model to ignore important contextual details. To address this, we first analyze the underlying cause, revealing that the semantic drift responsible for SCP is not random but is concentrated within a specific low-dimensional subspace. We also discover that the personalization process perturbs the embedding of the original base concept, making it an unstable reference point. Based on these insights, we introduce Test-time Embedding Adjustment with Adaptive Subspace Projection (AdaptSP), a training-free method that uses the stable, pre-trained embedding as an anchor. AdaptSP isolates the semantic drift and projects it onto the identified subspace, performing a precise adjustment that mitigates SCP while maintaining the subject identity. Our experiments show that this targeted approach significantly improves prompt fidelity and contextual alignment. 1 Introduction Text-to-image (T2I) diffusion models, built on denoising diffusion and score-based formulations, now synthesize highly realistic images with strong text alignment at scale ho2020denoising ; song2020score ; saharia2022imagen ; rombach2022high . These advances have fueled growing interest in generative personalization: adapting a pretrained T2I model to a user-specific concept (e.g., a person, pet, or object) from only a few reference images, while retaining the ability to place that concept into novel contexts via natural-language prompts gal2022image ; ruiz2023dreambooth . The core objective is to preserve the unique identity of the personal concept while remaining faithful to the prompt’s semantics. Despite rapid progress, misalignment between the generated image and the input prompt remains a major challenge. A robust generative personalization method should allow the user-defined visual concept to be composed with arbitrary contexts in the text prompt without losing fidelity or expressiveness. However, existing approaches often struggle to maintain prompt and generated image alignment, particularly with complex or multi-concept prompts kong2024omg ; zhu2025multibooth . This misalignment has commonly been attributed to language drift or overfitting ruiz2023dreambooth , arising from optimising on a limited number of reference images. Additional factors include limited expressiveness of textual embeddings zhang2023adding ; mou2024t2i and entangled reference sets avrahami2023break ; jin2024image , and semantic collapse bui2025mitigating , where the learned personalized token loses its original textual semantics and over-dominates the prompt. Prior work mitigates these issues through regularization strategies qiu2023controlling ; arar2024palp ; motamed2024lego or test-time embedding adjustment, which modifies the magnitude and direction of pretrained embeddings at inference to reduce semantic collapse bui2025mitigating . Compared to training-based regularization, test-time adjustment is lightweight, requires no retraining, and generalizes across personalization methods. However, it applies a global correction to the entire embedding and relies on post-personalization class embeddings as reference, which may introduce bias. It also overlooks structured, context-consistent residual drift, resulting in coarse adjustments for complex, multi-entity prompts. To address these limitations, we propose a more targeted test-time adjustment method with Adaptive Subspace Projection (AdaptSP) . Our approach is motivated by a simple decomposition: a personalized prompt embedding can be viewed as a class-based prompt embedding plus a concept-specific drift term that captures the identity of the personalised concept. This drift can be extracted as the residual embedding between the personalized prompt and its original class-based prompt. Our preliminary analysis shows that these residuals are stable across diverse contexts and concentrate in a low-dimensional subspace. AdaptSP leverages this structure to decouple identity from context by conditioning on a class prompt while injecting only a projected residual. By correcting only the structured drift component, AdaptSP preserves subject identity while restoring prompt fidelity in complex, multi-entity descriptions. Our proposed method is lightweight, training-free, and compatible with diverse personalization methods. Across two evaluated datasets, it consistently enhances text–image alignment without compromising subject fidelity in most settings. In summary, our contributions are as follows: • We show that residual embeddings are highly consistent across diverse prompts, suggesting they primarily encode context-invariant subject identity. • We represent the personalized concept as a residual embedding added to a class-based prompt embedding and introduce AdaptSP, a training-free, test-time method that decouples identity from context. This targeted adjustment mitigates prompt domination in the semantic collapsing problem and improves compositional fidelity in complex scenes. • Through extensive experiments with complex prompts, we demonstrate consistent gains in prompt alignment across standard personalization methods, supported by ablation studies and analysis of subject-fidelity metrics. 2 Related Work Generative Personalization. Generative personalization aims to bind a user-defined concept to a model for generation in new contexts. Classic subject-driven approaches include Textual Inversion (TI) gal2022image and DreamBooth ruiz2023dreambooth , and subsequent work explores parameter-efficient or targeted updates such as Custom Diffusion kumari2023multi , SVDiffusion han2023svdiff , OFT qiu2023controlling , and key-locking or rank-constrained tuning tewel2023key . Multi-concept and compositional personalization has been studied in methods like MultiBooth and related multi-subject settings zhu2025multibooth ; kong2024omg . Beyond subject-driven personalization, person/identity-driven methods focus on facial identity and composition, including FastComposer, Face0, DreamIdentity, and PhotoVerse xiao2024fastcomposer ; valevski2023face0 ; chen2024dreamidentity ; chen2023photoverse . Style-driven personalization explores controllable stylistic transfer, e.g., StyleDrop, StyleCrafter, and ArtAdapter sohn2023styledrop ; liu2023stylecrafter ; chen2024artadapter . Image- or adapter-based conditioning has also been explored to reduce tuning cost, such as IP-Adapter, PhotoMaker, and InstantID ye2023ipadapter ; li2023photomaker ; wang2024instantid , which inject image-derived identity cues through lightweight modules. Challenges in Personalization. Personalization from a few examples faces several recurring challenges: language drift and overfitting, limited expressiveness of the conditioning signal, entanglement of concepts in reference sets, and prompt misalignment. Overfitting is commonly mitigated through prior preservation and regularization ruiz2023dreambooth ; han2023svdiff ; qiu2023controlling ; tewel2023key . Limited expressiveness motivates adding new conditioning channels or adapters (e.g., ControlNet/T2I-Adapter/SCEdit) zhang2023adding ; mou2024t2i ; jiang2024scedit . Entanglement is addressed via masks, segmentation, or data augmentation avrahami2023break ; jin2024image ; safaee2024clic ; li2023generate ; chen2023disenbooth . Prompt misalignment and compositionality issues are studied in methods such as PALP, LEGO, and ReVersion arar2024palp ; motamed2024lego ; huang2024reversion . Test-time Adjustment. Test-time adjustment aims to align both the magnitude and direction of personalized embeddings with their original semantic concepts during inference, thereby mitigating semantic collapse bui2025mitigating . Prior work performs a global correction over the full prompt-embedding space using spherical linear interpolation (SLERP), interpolating between a personalized prompt embedding and its corresponding prior class embedding. This approach implicitly assumes that the undesired semantic drift is spread across the entire latent manifold and varies with each input prompt. 3 Semantic Collapsing in Personalized Generation Problem setup. We consider personalized text-to-image generation where a pre-trained diffusion model is adapted to represent a new subject using a learned concept token V^{*} (e.g., sks man ). Let c denote the corresponding anchor class token (e.g., man ). Given a class prompt p_{c} that contains c (e.g., a man sitting on a red chair ), we construct the personalized prompt p_{V^{*}} by replacing c with the learned token V^{*} while keeping the remaining text unchanged (e.g., a sks man sitting on a red chair). The text encoder \tau_{\phi} maps each prompt into a sequence of token embeddings, which are then fed to the U-Net via cross-attention during the denoising process. Semantic collapse in personalized tokens. Prior analyses bui2025mitigating indicate that the personalized token can

Read the complete text