Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Models

Haoming Wang & Wei Gao · 2025

Read Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Models by Haoming Wang & Wei Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-01932.txt.

Introduction

Haoming Wang and Wei Gao explain, in words, how a personalized image generator has been personalized. Such models are tuned to individual users' tastes, but users cannot see which aspects changed or by how much; visual examples alone are hard to read. Existing language explanations are too coarse to separate the multiple aspects of personalization or grade their levels.

Their technique, FineXL, produces natural language descriptions of each distinct aspect of personalization together with a quantitative score for its level. Experiments across several personalization scenarios and multiple kinds of image generation models showed FineXL improving the accuracy of explanations by about 56 percent over existing approaches. The work targets trust and control: if a system adapts itself to a user, the user deserves a precise account of the adaptation.

Catalog record

CREATORHaoming Wang & Wei Gao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-01932

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Models Haoming Wang & Wei Gao Abstract Image generation models are usually personalized in practical uses in order to better meet the individual users’ heterogeneous needs, but most personalized models lack explainability about how they are being personalized. Such explainability can be provided via visual features in generated images, but is difficult for human users to understand. Explainability in natural language is a better choice, but the existing approaches to explainability in natural language are limited to be coarse-grained. They are unable to precisely identify the multiple aspects of personalization, as well as the varying levels of personalization in each aspect. To address such limitation, in this paper we present a new technique, namely FineXL , towards Fine -grained e X plainability in natural L anguage for personalized image generation models. FineXL can provide natural language descriptions about each distinct aspect of personalization, along with quantitative scores indicating the level of each aspect of personalization. Experiment results show that FineXL can improve the accuracy of explainability by 56%, when different personalization scenarios are applied to multiple types of image generation models. 1 Introduction Open-sourced image generation models [ 6 ], including diffusion models [ 19 ; 31 ; 46 ; 49 ], generative adversarial networks (GANs) [ 16 ; 41 ] and auto-regressive models [ 47 ; 60 ], have been widely used in various applications, ranging from image stylization [ 30 ; 56 ], artistic content creation [ 27 ; 33 ], to virtual character design [ 28 ; 55 ]. In these applications, to meet the different users’ personalized needs, pre-trained models are usually fine-tuned with stylized data to derive personalized models [ 53 ; 26 ; 20 ; 29 ]. A large quantity of such personalized models have been made available online 1 1 1 On HuggingFace, there are > 80,000 personalized text-to-image models made and uploaded by individuals. for others to use, but most of these models offer limited explainability regarding how they are being personalized. More specifically, being different from the current practices in explainable AI that explore the correlation between model’s input, output and intermediate features [ 14 ; 7 ], such explainability interprets how an image generation model’s output changes after personalization [ 22 ], in aspects such as specific subjects or styles. For example, a personalized model could generate more stylized images in different ways (e.g., more vivid, artistic, historic, etc). Details about the fine-tuning process, such as the statistical properties of the training data, could provide useful information about explainability, but are also inadequately documented or even missing in the online repositories of most published models online. Figure 1: Importance of explainability for personalized image generation models to be used by human users Such explainability is important to real-world usage of those personalized image generation models published online. As shown in Figure 1 , many users nowadays do not opt to personalize models by themselves with their own datasets, but instead select the most suitable model from those personalized models available online. In these cases, users rely on clear and interpretable explanations for model selection . Without such explanations, users will have to manually prompt each model and empirically compare different models’ outputs, which is time-consuming or even infeasible considering the large number of available models. On the other hand, even for well-documented and close-sourced image generation models, such explainability could also help the model publisher’s fine-tuning process, as it offers insights into each fine-tuned model’s outputs and helps adjust the training data if the fine-tuned model’s behavior diverges from expectation. Figure 2: Personalized image generation in multiple aspects with varying levels Intuitively, mechanistic AI explainability [ 5 ; 15 ] can be applied to image generation models, and interpret the functionality of different neurons by learning interpretable features using another model, such as a sparse autoencoder [ 8 ]. However, these approaches are expensive and are limited to specific types of model structures [ 59 ] or labeled data [ 4 ; 32 ]. Alternatively, techniques of visual concept discovery can be used to identify the difference in model’s outputs before and after personalization through visual features [ 21 ; 63 ], but it is still difficult for human users to explicitly summarize such differences. As shown in Appendix F, when the model is personalized to generate images with more motion blur, images with more motion blur can be computationally identified as significant [ 63 ], but the commonality among these images may not be identified as “more motion blur” in human eyes. Figure 3: FineXL: fine-grained explanations of a personalized image generation model Instead, a better way is to explain the difference of personalized model’s output distribution in natural language , by leveraging vision-language models (VLMs) for image understanding and interpretation [ 3 ]. One can obtain natural language explanations of how models are personalized via image captioning [ 10 ; 66 ], or prompting a VLM to summarize the differences between images generated by the pre-trained model and personalized model [ 3 ]. Such summarization can be scaled to an unlimited number of comparisons [ 23 ; 71 ], ensuring that explanation is not biased. The key limitation of these techniques is that the natural language explanations they provide are coarse-grained . In most cases, an image generation model is personalized in multiple aspects, depending on the training data being used. The VLM’s summarization, when being used as the explanation of personalized models, is mostly vague and cannot precisely distinguish one aspect from others. For example, when personalization covers both aspects of vividity and abstractionism as shown in Figure 2 , the VLM summarization could be “a modern artistic style”, from which the two aspects cannot be clearly identified. Similarly, personalization in one aspect could also be done at varying levels [ 17 ], which cannot be precisely quantified by the existing techniques. To address these limitations, in this paper, we present a new technique, namely FineXL , towards Fine -grained e X plainability in natural L anguage for personalized image generation models. More specifically, as shown in Figure 3 , we aim to explicitly identify each aspect of how the model is being personalized and provide natural language explanation, along with quantitative scores indicating the level of each aspect of personalization. To achieve this objective, in the design of FineXL, we first use an image encoder (e.g., the CLIP encoder [ 44 ]) to map the divergence between output distributions of the personalized model and the pre-trained model into a high-level representation in a semantically rich representation space, to make the divergence is quantifiable. Then, to further interpret this representation, we leverage a VLM with sufficient representation power (e.g., GPT-4o) to generate multiple low-level natural language concepts, and convert them into vectors in the same representation space. To ensure that these low-level concepts correctly correspond to different aspects of personalization, we only select those concepts whose corresponding vectors are orthogonal in the embedding space. In this way, based on the linear representation hypothesis [ 61 ; 35 ], the high-level representation of the distribution divergence is compositional [ 57 ; 62 ], i.e., vectors of

Read the complete text