Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings
Azim Ospanov, Mohammad Jalali, Farzan Farnia · 2024
Read Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings by Azim Ospanov, Mohammad Jalali, Farzan Farnia (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18645.txt.
Introduction
Azim Ospanov, Mohammad Jalali, and Farzan Farnia build a score for how varied a text-to-image model's outputs really are in a 2024 paper. The standard similarity score checks whether one image matches its prompt, but says nothing about diversity across many generations. A model that always draws the same picture can score well while being useless creatively.
Their method splits the joint image-text statistics into a prompt-driven part and a remainder, using a matrix operation called the Schur complement. The entropy of the remainder becomes the Scendi score, a prompt-aware diversity measure. The same decomposition can mute a prompt's influence on an image's embedding to refocus analysis. Tests on image and text generators show the score tracks intrinsic diversity well. Code is available.
Catalog record
| CREATOR | Azim Ospanov, Mohammad Jalali, Farzan Farnia |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-18645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings Azim Ospanov, Mohammad Jalali, Farzan Farnia Abstract The use of CLIP embeddings to assess the fidelity of samples produced by text-to-image generative models has been extensively explored in the literature. While the widely adopted CLIPScore, derived from the cosine similarity of text and image embeddings, effectively measures the alignment of a generated image, it does not quantify the diversity of images generated by a text-to-image model. In this work, we extend the application of CLIP embeddings to quantify and interpret the intrinsic diversity of text-to-image models, which are responsible for generating diverse images from similar text prompts, which we refer to as prompt-aware diversity. To achieve this, we propose a decomposition of the CLIP-based kernel covariance matrix of image data into text-based and non-text-based components. Using the Schur complement of the joint image-text kernel covariance matrix, we perform this decomposition and define the matrix-based entropy of the decomposed component as the Schur Complement ENtopy DIversity (Scendi) score, as a measure of the prompt-aware diversity for prompt-guided generative models. Additionally, we discuss the application of the Schur complement-based decomposition to nullify the influence of a given prompt on the CLIP embedding of an image, enabling focus or defocus of the embedded vectors on specific objects. We present several numerical results that apply our proposed Scendi score to evaluate text-to-image and LLM (text-to-text) models. Our numerical results indicate the success of the Scendi score in capturing the intrinsic diversity of prompt-guided generative models. The codebase is available at https://github.com/aziksh-ospanov/scendi-score . 1 Introduction Prompt-guided generative models, which generate data guided by an input text prompt, have gained significant attention in the computer vision community. In particular, text-to-image and text-to-video models, which create visual data based on input text, have found many applications and are widely used across various content creation tasks. Given the important role of prompt-guided generative AI models in numerous machine learning applications, their training and evaluation have been extensively studied in recent years. A comprehensive evaluation of these models, addressing fidelity and diversity, is essential to ensure their effectiveness and adaptability across different use cases. As the CLIP model [ 45 ] provides a joint representation for text and image data, the CLIP embeddings have been widely used to evaluate and interpret the performance of text-to-image models. By calculating the cosine similarity between the CLIP embeddings of text and image data, the CLIPScore [ 14 ] serves as a fidelity metric for measuring the alignment between the text and the generated image. While the CLIPScore and similar uses of CLIP embeddings focus on evaluating the quality of generated samples, these embeddings have not yet been applied to assess the diversity of data produced by text-to-image models. Existing evaluation frameworks typically address diversity scores for unconditional sample generation (considering no input prompt), such as Recall [ 50 , 28 ] , Coverage [ 39 ] , RKE [ 20 ] , and Vendi [ 9 ] , which assess image samples independently of the input text. However, the diversity of images generated by a text-to-image model depends both on the variety of input text data and on the model’s intrinsic diversity, driving the model to produce varied images in response to similar prompts. Therefore, existing diversity metrics for unconditional sample generation cannot differentiate between diversity arising from varied prompts, i.e., prompt-driven diversity , and diversity contributed by the model itself, i.e., model-driven diversity . In this work, we leverage CLIP embeddings to propose a framework for quantifying and interpreting the intrinsic diversity of both text-to-image and image captioning (image-to-text) models. The primary objective of our approach is to decompose the CLIP image embedding into a text-based component, influenced by the input text, and a non-text-based component, arising from the model’s inherent randomness. To achieve this goal, we extend the kernel matrix entropy measures, i.e., Vendi and RKE scores, to enable a prompt aware diversity measurement. To do this, we consider a kernel similarity function and focus on the kernel covariance matrix for the generated (text, image) pairs. Considering the matrix-based entropy of the Image-based kernel covariance component C_{II} , one obtains the existing prompt-unaware Vendi [ 9 ] and RKE [ 20 ] scores. In this work, we propose applying the Gaussian elimination approach, and then decompose the image sub-covariance matrix, C_{II} as follows: C_{II}=\underbrace{C_{II}-C_{IT}C_{TT}^{{}^{\text{\scriptsize$-1$}}}C^{{}^{\text{\scriptsize$\top$}}}_{IT}}_{\text{\rm Model-driven Covariance Component}}+\underbrace{C_{IT}\,C_{TT}^{{}^{\text{\scriptsize$-1$}}}\,C_{IT}^{{}^{\text{\scriptsize$\top$}}}}_{\text{\rm Prompt-driven Covariance Component}} Note that the above is the sum of the Schur Complement component, C_{II}-C_{IT}C_{TT}^{-1}C_{IT}^{\top} , which represents the model-driven diversity component, and the remainder term C_{IT}C_{TT}^{-1}C_{TI} , which captures the prompt-driven variety in the generated images. Extending the Vendi and RKE score to prompt-aware diversity evaluation, we define the matrix-based entropy of the Schur complement component, as Schur Complement ENtropy DIversity (Scendi) score. We highlight that Scendi is a measure of model-driven diversity in the prompt-guided generated data. Figure 2 : Example of the CLIP embedding decomposition for an input image of ”Pumpkin next to Ostrich” generated by SDXL [ 44 ] . ImageNet [ 7 ] samples with the highest CLIP similarity to the input image after removing the “pumpkin” direction from its CLIP embedding via the Schur complement (left) and after removing the “ostrich” direction (right). To illustrate the effectiveness of the Scendi score in distinguishing model-driven diversity from prompt-driven effects, we present a comparative example. Figure compares the Scendi diversity scores for two sets of samples generated by DALL-E 3 [ 40 ] . The left-side subfigure is designed to represent model-driven diversity, showing images of diverse-breed cats and dogs generated using similar prompts like “cat and dog read a text.” In contrast, the right-side subfigure highlights the effects of prompt-driven diversity, featuring same-breed cats and dogs across diverse-occasion prompts. The unconditional diversity metrics, Vendi and RKE, report higher diversity for the right-side case. In contrast, our proposed Scendi score, designed to isolate intrinsic model diversity from prompt-driven effects, indicates greater diversity in the left-side case. This example highlights the advantage of the prompt-aware diversity measurement by Scendi score over existing unconditional diversity metrics in capturing the model-driven diversity. We present several numerical applications of the proposed framework to evaluate and interpret standard text-to-image and image-captioning models. Our results on several simulated scenarios with known ground-truth diversity indicate that the proposed entropy metric correlates with the non-text-based diversity in images, capturing variation not attributable to the text prompt. Additionally, we show that the decomposed feature component can neutralize the influence of specific objects in the text prompt within the image embedding. Specifically, we use this decomposed feature to diminish the impact of visible text in images, reducing its effect on the embedding. We also demonstrate how the decomposed image embedding can enhance or reduce the focus on particular objects or styles within an image, which can have implications for downstream applications of CLIP embeddings when emphasizing specific elements. For instance, Figure 2 shows a generated image of an ostrich next to a pumpkin. We showcase the ImageNet samples that achieve the highest CLIPScore when compared to the modified image embeddings obtained by canceling the “pumpkin” and “ostrich” directions. This demonstrates that the modified embeddings effectively eliminate the influence of those unwanted objects. The main contributions of this