Unveiling Concept Attribution in Diffusion Models

Quang H. Nguyen, Hoang Phan, Khoa D. Doan · 2024

Read Unveiling Concept Attribution in Diffusion Models by Quang H. Nguyen, Hoang Phan, Khoa D. Doan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02542.txt.

Introduction

Quang Nguyen, Hoang Phan, and Khoa Doan open up image-generating diffusion models to ask how their parts jointly produce a concept, in a 2024 paper. Earlier detective work could find layers where knowledge sits, but not how components cooperate, or cancel each other, when a prompt asks for an object or style.

Their framework scores every parameter's contribution to generating a given concept. Beyond the expected positive contributors, it uncovers components that work against the concept, a type previous localization studies missed. That two-sided map enables two fast editing tricks at generation time. Erase suppresses a concept by removing its positive drivers; amplify boosts one by damping its opponents. Both leave other concepts intact. Experiments confirm the pinpointed components carry the claimed roles, suggesting a fuller account of how generative models know things.

Catalog record

CREATORQuang H. Nguyen, Hoang Phan, Khoa D. Doan
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02542

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unveiling Concept Attribution in Diffusion Models Quang H. Nguyen, Hoang Phan, Khoa D. Doan Abstract Diffusion models have shown remarkable abilities in generating realistic and high-quality images from text prompts. However, a trained model remains largely black-box; little do we know about the roles of its components in exhibiting a concept such as objects or styles. Recent works employ causal tracing to localize knowledge-storing layers in generative models without showing how other layers contribute to the target concept. In this work, we approach diffusion models’ interpretability problem from a more general perspective and pose a question: “How do model components work jointly to demonstrate knowledge?” . To answer this question, we decompose diffusion models using component attribution, systematically unveiling the importance of each component (specifically the model parameter) in generating a concept. The proposed framework, called C omponent A ttribution for D iffusion Model (CAD), discovers the localization of concept-inducing (positive) components, while interestingly uncovers another type of components that contribute negatively to generating a concept, which is missing in the previous knowledge localization work. Based on this holistic understanding of diffusion models, we introduce two fast, inference-time model editing algorithms, CAD-Erase and CAD-Amplify; in particular, CAD-Erase enables erasure and CAD-Amplify allows amplification of a generated concept by ablating the positive and negative components, respectively, while retaining knowledge of other concepts. Extensive experimental results validate the significance of both positive and negative components pinpointed by our framework, demonstrating the potential of providing a complete view of interpreting generative models. Our code is available here . Figure 1 : Overview of our framework. We show that there exist positive and negative components in diffusion that increase or decrease the probability of the target concept, respectively. Removing those components will have the reverse effect. 1 Introduction Recent developments in diffusion models [ 18 , 26 , 35 , 36 ] have greatly improved the synthesizing capabilities, including image quality and generating a wide range of knowledge. However, these models lack interpretability; we do not fully understand how they can achieve such impressive performance and how they can generate images from only simple text prompts. To investigate how generative models recall concepts, a recent line of work studies which components in the model store knowledge [ 27 , 2 ] . In language models, Meng et al. 2022 propose causal tracing to locate layers storing facts and reveal that knowledge is localized in middle-layer MLP modules. This method is later transferred to diffusion models in [ 2 ] , discovering the knowledge distributed hypothesis ; this hypothesis shows that, different from language models, knowledge is distributed amongst a set of UNet components and the first self-attention layer of the text-encoder. These approaches shed light on interpreting generative models, enabling more effective model editing [ 2 , 3 ] . Nevertheless, they only focus on coarse-grained components (i.e., layers) and knowledge storage – modules that are responsible for generating concepts, potentially ignoring more subtle properties in the generative models and other types of modules, respectively. This paper first poses a more general question: How do components in diffusion models contribute to a generated concept? We then introduce a framework that predicts the model behavior given the presence of each component based on an efficient linear counterfactual estimator [ 34 ] . Through this framework, called C omponent A ttribution for D iffusion Model (CAD), we advance the understanding of how model components activate concepts (e.g., objects, styles, or explicit contents) in diffusion models. In contrast to the prior work that focuses on the model’s layers, CAD allows analysis of more fine-grained components. Specifically, focusing on the most fine-grained components, i.e., the model’s parameters, CAD could also identify concept-inducing (or positive) components similar to knowledge storage; however, instead of the distributed hypothesis in layers [ 2 ] , CAD discovers the localization hypothesis — knowledge is localized in a small number of parameters. Surprisingly, besides the positive components, CAD also reveals the existences of components that contribute negatively to generating the target concept, which is missing in the previous studies. Ablating these components decreases or increases the probability of generating the corresponding knowledge. This holistic understanding of diffusion models enables a lightweight model editing capability, i.e., to remove (positive) or recall (negative) a concept. Figure 1 illustrates the proposed CAD framework. In summary, our contributions are: • We propose CAD, a comprehensive framework, that can compute the attribution scores of the diffusion model components based on an efficient and effective linear counterfactual predictor. • Utilizing CAD, we confirm the existence of the concept-inducing (positive) model’s parameters, while revealing their localized nature. CAD also uncovers the existence of another type of components – the concept-amplification (negative) components. • Leveraging these observations of localized positive and negative components, we develop two lightweight knowledge editing algorithms, CAD-Erase for concept erasing and CAD-Amplify for concept amplification, respectively, for diffusion models. • We analyze CAD and evaluate the effectiveness of the proposed editing algorithms with extensive experiments, demonstrating their practicality and effectiveness. 2 Related Works Interpreting Neural Networks. Several research has extensively studied the black-box mechanism of neural networks to explain their behaviors. A line of works [ 33 , 4 , 41 ] aim to visualize important regions in the input of classification models by inspecting the gradient of feature map activations. Sundararajan et al. 2017 formalize the problem of attributing the input and propose two axioms to design attribution methods. Fundamentally different from those studies, we seek to attribute the model components , specifically parameters, in diffusion models. Knowledge Localization. Previous work [ 16 , 8 ] investigates how components of language model store factual knowledge. Similarly, Shah et al. 2024 propose model attribution to analyze the impact of individual components in the image classification and language prediction task, allowing us to understand the relationship between the model components and the model output. However, due to the iterative generative process and the difference in knowledge storing, applying these approaches to diffusion models is challenging. Another line of research [ 2 , 3 , 17 , 27 , 38 , 7 , 47 ] utilizes causal analysis to identify critical layers where knowledge is predominantly stored. For instance, Meng et al. 2022 show that language models tend to store factual information in certain causal layers, and modifying these layers improves generalization and specificity. Similarly, Basu et al. 2023 , Basu et al. 2024 apply this technique to T2I Latent Diffusion variants, targeting specific text-encoder and U-Net layers to remove unwanted elements such as nudity or copyrighted styles. While these methods have shown successes in localizing knowledge, Hase et al. 2024 discover that editing non-causal layers can also modify stored facts in language models. This finding implies that causal analysis may answer a different question from model editing. Furthermore, these approaches inspect the activations , which are dependent on the input, whereas our work studies the parameters of the model. Another study [ 9 ] examines the weight space of customized diffusion models by creating a dataset of thousands of models, showing it behaves as an interpretable manifold for generating and modifying visual identities. In contrast, our work offers an efficient approach to studying the role of the model component.

Read the complete text