Mitigating Semantic Collapse in Generative Personalization with a Surprisingly Simple Test-Time Embedding Adjustment
Anh Bui 1 Trang Vu 1 Trung Le 1 Junae Kim 2 Tamas Abraham 2 Rollin Omari 2 Amar Kaur 2 Dinh Phung · 2025
Read Mitigating Semantic Collapse in Generative Personalization with a Surprisingly Simple Test-Time Embedding Adjustment by Anh Bui 1 Trang Vu 1 Trung Le 1 Junae Kim 2 Tamas Abraham 2 Rollin Omari 2 Amar Kaur 2 Dinh Phung (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-22685.txt.
Introduction
Anh Bui and co-authors fix a subtle failure in personalized image generation called semantic collapse. Their 2025 paper studies models that learn a new visual concept from a few photos, stored as an embedding. Over training, that embedding drifts until it swamps the rest of a prompt, so a request for the concept wearing glasses and playing guitar returns a plain portrait.
The team traces the problem to unconstrained optimization letting the embedding wander in size and direction. Their fix needs no retraining: at test time they adjust the embedding's magnitude and direction back toward its textual meaning. The adjustment works across several personalization methods and improves how well outputs match prompts. The authors offer it as a simple patch for a widespread annoyance in image tools.
Catalog record
| CREATOR | Anh Bui 1 Trang Vu 1 Trung Le 1 Junae Kim 2 Tamas Abraham 2 Rollin Omari 2 Amar Kaur 2 Dinh Phung |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-22685
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Semantic Collapse in Generative Personalization with a Surprisingly Simple Test-Time Embedding Adjustment Anh Bui 1 Trang Vu 1 Trung Le 1 Junae Kim 2 Tamas Abraham 2 Rollin Omari 2 Amar Kaur 2 Dinh Phung Abstract In this paper, we investigate the semantic collapsing problem in generative personalization, an under-explored topic where the learned visual concept ( V ∗ V^{*} ) gradually shifts from its original textual meaning and comes to dominate other concepts in multi-concept input prompts. This issue not only reduces the semantic richness of complex input prompts like "a photo of V ∗ V^{*} wearing glasses and playing guitar" into simpler, less contextually rich forms such as "a photo of V ∗ V^{*} " but also leads to simplified output images that fail to capture the intended concept. We identify the root cause as unconstrained optimisation, which allows the learned embedding V ∗ V^{*} to drift arbitrarily in the embedding space, both in direction and magnitude. To address this, we propose a simple yet effective training-free method that adjusts the magnitude and direction of pre-trained embedding at inference time, effectively mitigating the semantic collapsing problem. Our method is broadly applicable across different personalization methods and demonstrates significant improvements in text-image alignment in diverse use cases. Our code is anonymously published at https://anonymous.4open.science/r/Embedding-Adjustment . 1 Introduction Text-to-image (T2I) diffusion models have achieved unprecedented fidelity and flexibility in image generation and sparked growing interest in generative personalization [ 9 , 25 ] . This emerging problem aims to generate images of a specific user-defined visual concept (e.g. a particular person, pet, or object) in different contexts (e.g. on the beach) using a small set of user-provided reference images paired with text prompts describing the desired context. The core objective is to generate visually compelling images that faithfully preserve the unique characteristics of the personal concept while remaining semantically aligned with the textual prompt. Despite recent progress, misalignment between the generated image and the textual prompt is still a major concern. A robust generative personalization method should allow the user-defined visual concept to be composed with arbitrary contexts in the text prompt without losing fidelity or expressiveness. However, existing approaches often struggle to maintain prompt and generated image alignment, particularly with complex or multi-concept prompts [ 15 , 37 ] . In these scenarios, the introduced concept can overpower or distort other elements in the prompt, leading to unsatisfactory generations. This issue has commonly been attributed to language drift or overfitting [ 25 ] , arising from optimising on a limited number of reference images. Beyond overfitting, other factors have also been attributed, such as the limited expressiveness of textual embeddings which compress complex visual concepts into single tokens [ 36 , 20 ] and the entangled nature of reference sets, where samples may contain co-occurring objects or irrelevant contextual features [ 1 , 14 ] . In this work, we investigate an under-explored failure mode of generative personalization, which we term the semantic collapsing problem (SCP). SCP refers to the phenomenon where the learned personalized token for a new concept loses its original textual semantic meaning while acquiring increased visual information from the reference concept during the personalization process. As a result, when a user combines the personalized token with other descriptive context in a prompt, the model’s output is overwhelmingly biased towards the personalized concept, often neglecting the other intended elements in the prompt. For example, if one learns a token V^{*} to represent a particular dog, a prompt like “ V^{*} playing with a ball in a park” may yield an image that vividly depicts the dog but fails to properly render the ball or the park background. In essence, the prompt’s semantic richness collapses to a simpler form centred on V^{*} . While SCP does not severely affect trivial prompts (e.g., “a photo of V^{*} ”, where the entire prompt is just the concept itself), it undermines the compositionality of the T2I diffusion model on complex prompts. Notably, SCP is distinct from the language drift problem, which describes a different failure where the personalized model overfits to a learned concept and loses generalisation. In SCP, the issue arises when the personalized embedding no longer retains any meaningful textual semantics. We identify the root cause as unconstrained optimisation , which allows the learned embedding to drift arbitrarily in the embedding space, both in direction and magnitude. We propose a simple yet effective remedy: a training-free, test-time embedding adjustment strategy that realigns the learned concept embedding with its original semantic meaning at inference time. The key idea is to calibrate the embedding’s magnitude and direction to be closer to that of its reference concept. This adjustment is done without altering the model weights or requiring any additional training. The embedding is modified on-the-fly before image generation. By enforcing a small rotation and rescaling in the text encoder’s latent space, we constrain the personalized token to behave more like a regular word, ensuring that it contributes to the image generation in balance with other tokens in the prompt. Our approach is lightweight and broadly compatible with different personalization methods and demonstrates significant improvements in text-image alignment in diverse use cases. In summary, our contributions are as follows: ❶ We define the semantic collapsing problem (SCP) in generative personalization problem, and provide an empirical analysis of its existence in both textual and image spaces. Our analysis reveals its root cause, which is the unconstrained optimisation. ❷ We propose the test-time embedding adjustment, a novel solution that requires no additional training, to mitigate SCP by aligning the learned embedding’s direction and norm with its original semantic concept. ❸ We demonstrate that our proposed approach can be applied into different personalization methods such as Textual Inversion [ 9 ] , DreamBooth [ 25 ] , and Custom Diffusion [ 16 ] to significantly improve image generation in complex prompts across a wide range of scenarios. (a) Intermediate generated images (b) Alignment scores Figure 1 : Illustration of the SCP on Textual Inversion. Left: The intermediate generated images \hat{x} of a prompt ‘a photo of a V^{*} wearing glasses and writing in a red notebook ’. The generated image is gradually biased towards the personalized concept V^{*} (i.e., easier to recognize as ‘Henry Cavill’) and loses the context p (i.e., harder to recognize as ‘writing in a red notebook’) through out the personalization process. Right: The alignment scores (average over 100 random seeds) which empirically validate the SCP. The alignment S(\hat{x},p) ( \square ) with the context p drops over time, while the alignment S(\hat{x},x_{gt}) ( + ) with the ground truth x_{gt} increases. 2 Personalizing T2I Generative Models Given a text-to-image diffusion model \epsilon_{\theta} , where \epsilon_{\theta}(x_{t},t,p) represents the predicted noise at time step t given the textual embedding \tau(p) of a prompt p and the noisy intermediate vector x_{t} [ 11 , 29 , 24 ] . And given a set of personal images \mathcal{X}=\{x_{1},x_{2},\cdots,x_{n}\} and a pre-trained T2I model \epsilon_{\theta} , the goal of generative personalization is to identify a textual embedding v^{*} associated with a specific verbalizable keyword V^{*} (e.g., ‘sks’, ‘<new>’, etc.). This keyword represents the implicit visual concept shared in the reference set \mathcal{X} , enabling