Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder

Wonwoong Cho, Yan-Ying Chen, Matthew Klenk, David I. Inouye, Yanxia Zhang · 2025

Read Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder by Wonwoong Cho, Yan-Ying Chen, Matthew Klenk, David I. Inouye, Yanxia Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-11937.txt.

Introduction

Wonwoong Cho and co-authors add fine control knobs to text to image diffusion models in a 2025 paper. Prompt text steers pictures coarsely, but dialing a numeric trait like eye openness or car width, let alone several at once in a new domain, stays clumsy. Their plug in adapter learns those controls from sample images that need not be paired.

A decoupled attention design blends the learned traits with text guidance. A conditional variational autoencoder inside the adapter curbs overfitting and keeps outputs varied. On two public data sets, the adapter beat fine tuning baselines on control accuracy, range, and trait separation. It also topped style based methods while staying a single, scalable module.

Catalog record

CREATORWonwoong Cho, Yan-Ying Chen, Matthew Klenk, David I. Inouye, Yanxia Zhang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-11937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder Wonwoong Cho, Yan-Ying Chen, Matthew Klenk, David I. Inouye, Yanxia Zhang Abstract Text-to-Image (T2I) Diffusion Models have achieved remarkable performance in generating high quality images. However, enabling precise control of continuous attributes, especially multiple attributes simultaneously, in a new domain (e.g., numeric values like eye openness or car width) with text-only guidance remains a significant challenge. To address this, we introduce the Attribute (Att) Adapter , a novel plug-and-play module designed to enable fine-grained, multi-attributes control in pretrained diffusion models. Our approach learns a single control adapter from a set of sample images that can be unpaired and contain multiple visual attributes. The Att-Adapter leverages the decoupled cross attention module to naturally harmonize the multiple domain attributes with text conditioning. We further introduce Conditional Variational Autoencoder (CVAE) to the Att-Adapter to mitigate overfitting, matching the diverse nature of the visual world. Evaluations on two public datasets show that Att-Adapter outperforms all LoRA-based baselines in controlling continuous attributes. Additionally, our method enables a broader control range and also improves disentanglement across multiple attributes, surpassing StyleGAN-based techniques. Notably, Att-Adapter is flexible, requiring no paired synthetic data for training, and is easily scalable to multiple attributes within a single model. The project page is available at: https://tri-mac.github.io/att-adapter/ . 1 Introduction Large pretrained T2I diffusion models [ 31 , 30 , 27 , 34 ] have achieved remarkable success in generating high quality and realistic images based solely on text prompts. However, allowing users precise control over the generated content or tailoring it to a specific new domain remains challenging [ 8 , 11 , 32 , 43 ] . For example, with the prompt “A photo of a woman smiling”, one might want to adjust specific facial attributes such as “width of nose” and “openness of eyes” by assigning specific numeric values to these attributes. As Fig. 1 demonstrates, current models like Stable Diffusion [ 31 ] (Realistic Vision 4.0 1 1 1 https://huggingface.co/stablediffusionapi/realistic-vision-v40 ) fail to reliably control these fine-grained attributes, often misinterpreting or neglecting the specified modifications. Figure 1 : Att-Adapter enables continuous control (e.g., 0 to 1) over subtle visual details like nose width or eye openness, which are difficult to capture with discrete text-based conditioning. Early domain adaptation methods rely on text-based conditioning, either by optimizing text tokens [ 32 , 8 , 45 ] or fine-tuning T2I models using Low-Rank Adaptation (LoRA) [ 16 ] . However, text-only conditioning are inherently limited: texts often underspecifies visual details such as object type, perspective and style [ 17 , 45 ] , and existing methods typically handle only discrete attribute values. Recent efforts attempt to address continuos attribute [ 3 , 22 , 28 , 9 ] . AttributeControl [ 3 ] modifies CLIP embeddings but struggles with indescribable visual attributes (e.g., nose width) that rarely appear in text. Other methods, such as W_{+} Adapter [ 22 ] and PreciseControl [ 28 ] , depend on pretrained StyleGAN latent spaces, making generation quality highly dependent on separately trained GANs. ConceptSlider [ 9 ] proposes a LoRA-based controller for indescribable visual attributes but requires paired data and trains separate models per attribute (Fig. 2 ). This limits its use to the case where only one attribute is paired while others remain unchanged, which makes it infeasible in unpaired multi-attribute data settings without generating synthetic pairs. However, generating synthetic paired data adds extra preprocessing steps and restricts performance to the quality of the data generator (e.g., StyleGAN). A detailed comparison with existing methods is listed in Table 1 . Figure 2 : Workflow comparison. Unlike ConceptSlider [ 9 ] , which demands paired data and separate models for each attribute, our approach supports unpaired multi-attribute control in a single framework. See Sec. 2 for details. Pretrained GANs Continuous Attrs Indescribable Visual Attrs Multi-Attrs Training Real Data Training AttrCtrl [ 3 ] ✗ ✓ ✗ ✗ ✗ W_{+} Adapter [ 22 ] ✓ ✓ ✓ ✗ ✗ ConceptSlider [ 9 ] ✓ ✓ ✓ ✗ ✗ ITI-GEN [ 45 ] ✗ ✗ ✓ ✗ ✓ LoRA [ 16 ] ✗ ✗ ✓ ✓ ✓ Ours ✗ ✓ ✓ ✓ ✓ Table 1 : Requirement and capability comparisons . The leftmost column shows a requirement while the others show benefits. Multi-Attrs indicate many attributes such as 20. Most baselines rely on a pretrained latent concept space where the synthetic attribute values are not directly from the real-world data. In this paper, we propose Att-Adapter , a novel plug-and-play adapter that enables precise control over multiple attributes in pretrained diffusion models. As illustrated in the top row of Fig. 2 , Att-Adapter learns multi-attribute control from unpaired real data by leveraging a decoupled cross-attention module [ 44 ] to effectively fuse conditions from different modalities (e.g., view angle, size, style). We found that directly mapping multiple attributes with a naive MLP leads to overfitting (Fig. 8 and Table 5 ); To this end, we incorporate a Conditional Variational Autoencoder (CVAE) [ 38 , 48 ] to improve robustness against overfitting. By performing an approximate Bayesian inference [ 20 ] , the CVAE can naturally regularize Att-Adapter via its KL divergence term. This not only mitigates overfitting but also smooths the conditional representation, modeling the natural randomness for given attribute condition. Extensive experiments on face and car datasets demonstrate that Att-Adapter outperforms state-of-the-art methods in both continuous and discrete attribute control. Our contributions are as follows: • Unpaired Multi-Attribute Control: Att-Adapter simultaneously controls multiple attributes from unpaired data, eliminating the need for costly paired samples. • Continuous and Indescribable Attribute Manipulation: Our approach precisely controls subtle visual details that are difficult to describe textually. • Robust Control: The integration of a CVAE mitigates overfitting, ensuring stable and accurate attribute control. 2 Related Works Additional related works on controllable T2I models, such as IP-Adapter [ 44 ] are in Section B in the Appendix. Single Attribute Control for T2I Generation. Diffusion models (DMs) [ 6 , 15 , 37 ] are one of the widely-used text-to-image generative models due to its impressive performance. To customize DMs for a particular domain or new concept, fine-tuning the U-Net is a common approach [ 32 , 21 ] , although it can lead to catastrophic forgetting [ 25 , 24 ] . Instead of retraining all model parameters, Low-Rank Adaptations (LoRA) [ 16 ] , adds trainable low rank decomposition matrices to each layer of a frozen model, significantly reducing trainable parameters while maintaining performance. Other methods, such as DreamBooth [ 32 ] and Textual Inversion [ 8 ] , modify text embeddings of pretrained T2I models [ 32 , 8 , 45 ] . Both LoRA and text embedding approaches require training a new LoRA network or token for each attribute. Additionally, they struggle to represent continuous attribute values without discretization. Multi-Attribute Control for T2I Generation. Existing methods often combine individually trained LoRAs through weighted averaging, which can lead to concept vanish or distortion [ 24 , 21 , 43 , 11 ] . Approaches like Mix-of-Show [ 11 ] and Lora-composer [ 43 ] attempt to address this but still require separate LoRAs, limiting

Read the complete text