ANIMALBOOTH: MULTIMODAL FEATURE ENHANCEMENT FOR ANIMAL SUBJECT PERSONALIZATION
Chen Liu, Haitao Wu, Kafeng Wang, Xiaowang Zhang · 2025
Read ANIMALBOOTH: MULTIMODAL FEATURE ENHANCEMENT FOR ANIMAL SUBJECT PERSONALIZATION by Chen Liu, Haitao Wu, Kafeng Wang, Xiaowang Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-16702.txt.
Introduction
Chen Liu, Haitao Wu, Kafeng Wang, and Weiran Huang build a pet image generator in a 2025 paper. Making an image of a specific animal is hard. Animals vary a lot in shape, and fine marks set one apart. Past tools often lost those marks, so the output drifted from the real animal.
Their system sharpens features drawn from several input types at once. It holds identity better across varied scenes. The team also built AnimalBench, a high-grade dataset for this task. Tests show their method beats strong rivals on identity and image quality. Such tools could help owners and media makers place a known pet in new scenes.
Catalog record
| CREATOR | Chen Liu, Haitao Wu, Kafeng Wang, Xiaowang Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-16702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ANIMALBOOTH: MULTIMODAL FEATURE ENHANCEMENT FOR ANIMAL SUBJECT PERSONALIZATION Chen Liu, Haitao Wu, Kafeng Wang, Xiaowang Zhang Abstract Personalized animal image generation is challenging due to rich appearance cues and large morphological variability. Existing approaches often exhibit feature misalignment across domains, which leads to identity drift. We present AnimalBooth , a framework that strengthens identity preservation with an Animal Net and an adaptive attention module, mitigating cross domain alignment errors. We further introduce a frequency controlled feature integration module that applies Discrete Cosine Transform filtering in the latent space to guide the diffusion process, enabling a coarse to fine progression from global structure to detailed texture. To advance research in this area, we curate AnimalBench , a high resolution dataset for animal personalization. Extensive experiments show that AnimalBooth consistently outperforms strong baselines on multiple benchmarks and improves both identity fidelity and perceptual quality. † † address: 1 Tianjin University 2 Tsinghua University {ttjy, wuhaitao, xiaowangzhang}@tju.edu.cn, [email protected] 1 INTRODUCTION Personalized multimodal generation is a prominent yet challenging subfield that aims to synthesize images conforming to both textual descriptions (text–image consistency) and the intrinsic characteristics of custom concepts (identity consistency) [ 1 , 2 , 3 , 4 ] . This paradigm shows strong potential across diverse applications ranging from creative artistry to product design [ 5 , 6 ] . However, accurately capturing the identity of animal subjects, which exhibit complex visual features such as fine-grained fur textures and non-rigid morphologies, remains a significant hurdle [ 7 , 8 ] . Figure 1: Examples of Animal-HD dataset. Each instance provides five components: (a) a high-quality source image, (b) a detailed caption describing the scene and subject, (c) a precise pixel-level semantic segmentation mask of the primary subject, (d) the resulting masked subject for visual verification, and (e) the corresponding paired image. Existing personalization methodologies can be broadly categorized by their feature extraction and optimization strategies. The first category comprises optimization-based methods that achieve high fidelity by fine-tuning model components. A representative example is DreamBooth [ 9 ] , which fine-tunes the UNet backbone, while Textual Inversion [ 10 ] instead optimizes word embeddings. This paradigm has been extended to support multiple customized concepts [ 11 , 12 , 13 ] , including applications such as virtual dressing and editing [ 14 ] . Despite their fidelity, such approaches demand substantial computational resources and often exhibit an inherent trade-off between fidelity and diversity due to overfitting. The second category, encoder-based and tuning-free methods, offers a more efficient alternative. For instance, IP-Adapter [ 15 ] and its successors [ 16 ] utilize frozen external encoders (e.g., CLIP) to inject identity features. While efficient, their performance is fundamentally limited by the representational capacity of the encoder, particularly when modeling animals with unique identity cues such as body structures and fur patterns. Consequently, identity distortion and detail loss are frequent. Other directions include retrieval-augmented generation [ 17 ] or LLM-enhanced frameworks [ 18 , 19 , 20 ] , yet the challenge of inadequate domain-specific representation remains. Motivated by bridging this domain gap, we propose AnimalBooth , a feature-enhanced, fine-tuning-free personalized generation framework specifically tailored for animals. Our approach adopts a dual-enhancement strategy: at the macroscopic level, we employ an Animal-Net with adaptive attention to reinforce structural and identity understanding; at the microscopic level, we introduce a frequency-controlled feature integration module, where Discrete Cosine Transform (DCT) filtering guides the diffusion process to preserve fine textures such as fur. Furthermore, while numerous datasets exist for general consistency, a dedicated benchmark for animal personalization is lacking. To address this, we construct AnimalBench , a high-resolution dataset for animal personalization, which will be released upon acceptance. Our contributions are threefold: (1) We propose a dedicated branch architecture comprising a lightweight Animal-Net and adaptive attention, effectively mitigating identity distortion. (2) We design a frequency-controlled feature integration module leveraging DCT filtering in latent space to enhance attribute manipulation and texture fidelity. (3) We establish and release AnimalBench, a high-definition dataset for animal personalization, on which our method achieves state-of-the-art performance. Figure 2: Qualitative comparison with other SOTA methods. Figure 3: AnimalBooth primarily consists of a trainable Animal-Net and a frozen Photography-Net. The Animal-Net incorporates an Adaptive Attention module for efficient identity feature injection and a Frequency-Controlled module for enhanced control over visual attributes like structure and texture, while the Photography-Net integrates these features with text prompts within the latent space. 2 METHODOLOGY 2.1 Overall Architecture As depicted in Fig. 3 , AnimalBooth effectively integrates a trainable Animal-Net with a frozen Photography-Net. The Animal-Net directly refines the complex features and flexible morphologies of animals within the feature space, effectively bypassing the complexities of cross-domain alignment. Through adaptive attention, identity features are efficiently injected. The incorporation of a frequency-controlled module further enhances fine-grained control over visual attributes such as the structure and textural details of the generated images. The frozen Photography-Net then merges these extracted fine-grained features with text prompts in the latent space. 2.2 Animal-Net In the task of personalized animal image generation, precise extraction of fine-grained animal identity features is paramount for maintaining consistency between generated images and reference animals. To achieve this, we propose a specialized Animal-Net capable of simultaneously capturing both semantic information and high-frequency textural features of animals. Specifically, given an animal reference image \mathcal{X}_{a}\in\mathbb{R}^{3\times H\times W} , we first transform it into a latent space representation \mathbf{L}_{a}\in\mathbb{R}^{4\times\frac{H}{8}\times\frac{W}{8}} using a frozen VAE encoder [ 21 ] . We then extract token embeddings from \mathcal{X}_{a} using a frozen CLIP image encoder [ 22 ] and a trainable projection layer. Here, we employ Q-Former [ 23 ] as the projection layer. Subsequently, animal features within the Animal-Net interact extensively through a cross-attention mechanism, similar to the interaction between text and image features in the original Text-to-Image (T2I) model [ 21 ] . This interaction ensures a deep fusion of semantic and textural features. Finally, the output of the Animal-Net is aligned in parallel with the Photography-Net, injecting these fine-grained animal identity features into the Photography-Net via an adaptive attention module. Notably, the Animal-Net is solely used for encoding reference images; therefore, during the diffusion process, no noise is added to the reference image, and it undergoes only a single forward pass. 2.3 Adaptive Attention Module To enable personalized animal image generation, the Photography-Net must possess both its original generative capabilities and the ability to fuse animal identity features. We freeze the core modules of the Photography-Net to preserve the former, and achieve the latter through an adaptive attention module. The architecture of the Photography-Net in AnimalBooth build upon SD v1.5 [ 21 ] , with all self-attention modules replaced by adaptive attention modules. As shown in Fig. 3 , an adaptive attention module consists of a frozen self-attention module and a learnable cross-attention module. Its self-attention weights are initialized from SD v1.5 to retain generative capacity. Given the query features \mathbf{Z}_{n} from the Photography-Net and the animal identity features \mathbf{F}_{a} from the Animal-Net, the output \mathbf{O}_{h} of the adaptive attention module is defined as: \mathbf{O}_{h}=\underbrace{\operatorname{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}\right)\mathbf{V}}_{\text{Self-attention}}+\lambda\underbrace{\operatorname{Softmax}\left(\frac{\mathbf{Q}\left(\mathbf{K}_{ID}\right)^{\top}}{\sqrt{d}}\right)\mathbf{V}_{ID}}_{\text{Cross-attention}}~, (1) where \lambda\in[0,1] controls the strength of the animal identity feature condition. \mathbf{Q} , \mathbf{K} , \mathbf{V} are derived from \mathbf{Z}_{n} , while \mathbf{K}_{ID}=\mathbf{F}_{a}\mathbf{W}_{k_{ID}} and \mathbf{V}_{ID}=\mathbf{F}_{a}\mathbf{W}_{v_{ID}} are derived from \mathbf{F}_{a} . \mathbf{W}_{q} , \mathbf{W}_{k} ,