Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control
Basim Azam, Naveed Akhtar · 2025
Read Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control by Basim Azam, Naveed Akhtar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18324.txt.
Introduction
Basim Azam and Naveed Akhtar present a computing paper from 2025 on image generators that take text prompts. Such models can make unfair or unsafe images, so guardrails are a must. Past guardrails handle one concern at a time and hide how they work. Many also require changes to the model itself, which can hurt quality.
Their method adds a plug-in unit that learns a control space for many concerns at once. The space is built to stay readable, so users can see what each control does. It can plug in at two points: where text is encoded, or inside the image-building core. Tests showed strong control of fairness and safety themes without retraining the base model. The authors say the design scales to fresh concerns as they arise.
Catalog record
| CREATOR | Basim Azam, Naveed Akhtar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18324
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Plug-and-Play Interpretable Responsible Text-to-Image Generation via Dual-Space Multi-facet Concept Control Basim Azam, Naveed Akhtar Abstract Ethical issues around text-to-image (T2I) models demand a comprehensive control over the generative content. Existing techniques addressing these issues for responsible T2I models aim for the generated content to be fair and safe (non-violent/explicit). However, these methods remain bounded to handling the facets of responsibility concepts individually, while also lacking in interpretability. Moreover, they often require alteration to the original model, which compromises the model performance. In this work, we propose a unique technique to enable responsible T2I generation by simultaneously accounting for an extensive range of concepts for fair and safe content generation in a scalable manner. The key idea is to distill the target T2I pipeline with an external plug-and-play mechanism that learns an interpretable composite responsible space for the desired concepts, conditioned on the target T2I pipeline. We use knowledge distillation and concept whitening to enable this. At inference, the learned space is utilized to modulate the generative content. A typical T2I pipeline presents two plug-in points for our approach, namely; the text embedding space and the diffusion model latent space. We develop modules for both points and show the effectiveness of our approach with a range of strong results. Our code can be accessed at https://basim-azam.github.io/responsiblediffusion/ Figure 1 : Our unique plug-and-play interpretable approach simultaneously controls a range of concepts for responsible and fair image generation with text-to-image pipelines. Our method enables control over both text embedding space and latent diffusion space. Shown examples compare unfair and unsafe generation for the given prompts by the Stable Diffusion (top), along with their responsible counterparts resulting from our approach influencing the text encoder (middle) and the diffusion model (bottom). We control individual concepts for diverse/safe generation in these examples while modeling them in continuous composite responsible semantic spaces. 1 Introduction Rapid advances in text-to-image (T2I) generative pipeline are revolutionizing numerous vision applications, offering unprecedented convenience for synthesizing high quality visual content using textual descriptions Rombach et al. 2022b ; Ramesh et al. 2022 ; Peebles and Xie 2023 ; Karras et al. 2022 ; Ho et al. 2020 . Unfortunately, this convenience coupled with the generated content realism offered by publicly accessible T2I models, e.g., Stable Diffusion Rombach et al. 2022b , DALL-E Ramesh et al. 2022 , Imagen Saharia et al. 2022 , also has undesired ethical implications. Confronting the core values of fair and safe (non-violent/explicit) development and deployment of Artificial Intelligence, these models lack in producing generative content responsibly Gu et al. 2023 ; Zhang et al. 2025 . The reasons of irresponsible content generation by contemporary T2I models lie deep in their training process, which must use large volumes of (mostly uncurated) data. Issues like inappropriate content and harmful stereotyping lurking in the data are passed on to the generative models through training Ba et al. 2023 ; Tsai et al. 2023 , which cause problems after their deployment. Hence, it is critical to develop methodologies to explicitly control generative models from producing irresponsible content. Addressing that, several techniques have emerged recently, encompassing approaches like input prompt filtering Yoon et al. 2024 ; Ba et al. 2023 , post-hoc content moderation Yang et al. 2024 ; Wang et al. 2024 ; Kumari et al. 2023 , machine unlearning Wu et al. 2024 ; Park et al. 2024 ; Huang et al. 2023 ; Zhang et al. 2024a , and model editing Gandikota et al. 2024 ; Xiong et al. 2024 . Though effective, these approaches face limitations like requiring human-intervention, ad-hoc methodology, and selective dealing of different facets of the concepts defining ‘responsible generation’. Another contributing factor to the ethical concerns related to T2I models is our current lack of wholesome understanding of the latent space of the diffusion models. Researchers have already started exploring interpretability Haas et al. 2024 ; Zhang et al. 2024b and manipulation of diffusion model latent spaces to mitigate unintended T2I model behavior Gandikota et al. 2024 ; Li et al. 2024 . Controlling the latent space holds promise when it is possible to disentangle semantic concepts in it. One recent inspiring approach in this direction Li et al. 2024 identifies vectors in the latent diffusion space in the directions of individual semantic concepts, which can be used to restrict model outputs. However, it is arguable that such a discretized treatment of the concepts within the diffusion latent space is restrictive. It is also intrinsically limited to dealing with diverse facets of the concepts individually. Overall, it still remains a widely open challenge for the research community to comprehensively embed the multi-faceted concepts related to fair and safe image generation in T2I pipelines. In this work, we address this by presenting a unique scalable approach to extensively incorporate fair and safe generative abilities in a T2I pipeline in a plug-and-play manner. Our key insight is that a T2I model can be distilled for a range of user-desired responsibility concepts, conditioned on the original model. We perform this distillation by inducing a student model that treats the concepts related to fairness and safety in a continuous space. Within this space, we further leverage concept whitening Chen et al. 2020 to disentangle the representations of concepts for a better ultimate control on the generative image modulation. The underlying framework of our technique is generic, in that it is applicable to any representation space encoding semantic concepts. Hence, we explore its application to both text encoder embedding space and diffusion model latent space in the T2I pipeline. Methods for both variants address their unique challenges, but prove equally effective - see Fig. 1 . The key contributions of this work are summarized below. • We propose a unique plug-and-play method for responsible image generation with T2I pipeline that enables comprehensive incorporation of fairness and safety concepts while modeling them under a distilled continuous space conditioned on the generative model. • We tailor our method as a text embedding plug-in, and diffusion latent plug-in while also leveraging concept whitening to ensure precise interpretable control over the content. Both plug-ins are employed in our approach. • With extensive experiments, we not only demonstrate state-of-the-art or comparable performance in fair and safe image generation, but also show other unique interesting properties of our approach. 2 Related Works Text-to-image (T2I) generation has significantly expanded the capabilities of generative AI, enabling high-quality outputs using textual descriptions Rombach et al. 2022b ; Ramesh et al. 2022 ; Peebles and Xie 2023 ; Karras et al. 2022 ; Ho et al. 2020 . While T2I methods like Stable Diffusion Rombach et al. 2022b , DALL-E Ramesh et al. 2022 , GLIDE Nichol et al. 2021 , and Imagen Saharia et al. 2022 have achieved impressive realism and adaptation in numerous applications, they also face several challenges. One of them is the concern about their irresponsible and unethical content generation Gu et al. 2023 ; Ba et al. 2023 . Contemporary T2I models are known to generate inappropriate images, including nude and violent content, and they are also susceptible to social and cultural biases persistently found in the public domain datasets used to train these models Cho et al. 2023 ; Luccioni et al. 2024