T a x a D i f f u s i o n : Progressively Trained Diffusion Model for Fine-Grained Species Generation

Amin Karimi Monsefi, Mridul Khurana, Rajiv Ramnath, Anuj Karpatne, [2pt] Wei-Lun Chao, Cheng Zhang, [6pt] The Ohio State University Virginia Tech Texas A&M University · 2025

Read T a x a D i f f u s i o n : Progressively Trained Diffusion Model for Fine-Grained Species Generation by Amin Karimi Monsefi, Mridul Khurana, Rajiv Ramnath, Anuj Karpatne, [2pt] Wei-Lun Chao, Cheng Zhang, [6pt] The Ohio State University Virginia Tech Texas A&M University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01923.txt.

Introduction

Amin Karimi Monsefi and co-authors present TaxaDiffusion in a 2025 paper on making images of animal species. Species that share roots look close, with gaps in fine traits of shape, pattern, and color. Standard image makers treat each species as its own box and miss those links.

Training climbs the tree of life step by step, from broad class down to species. Broad traits shared by kin are learned first, then fine gaps are sharpened at each lower rung. That order lets the model learn from kin when samples for one species are few. Tests on three fine grained animal sets beat past methods on image faithfulness. The project page shows samples and code links for the work.

Catalog record

CREATORAmin Karimi Monsefi, Mridul Khurana, Rajiv Ramnath, Anuj Karpatne, [2pt] Wei-Lun Chao, Cheng Zhang, [6pt] The Ohio State University Virginia Tech Texas A&M University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01923

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

T a x a D i f f u s i o n : Progressively Trained Diffusion Model for Fine-Grained Species Generation Amin Karimi Monsefi, Mridul Khurana, Rajiv Ramnath, Anuj Karpatne, [2pt] Wei-Lun Chao, Cheng Zhang, [6pt] The Ohio State University Virginia Tech Texas A&M University Abstract We propose TaxaDiffusion, a taxonomy-informed training framework for diffusion models to generate fine-grained animal images with high morphological and identity accuracy. Unlike standard approaches that treat each species as an independent category, TaxaDiffusion incorporates domain knowledge that many species exhibit strong visual similarities, with distinctions often residing in subtle variations of shape, pattern, and color. To exploit these relationships, TaxaDiffusion progressively trains conditioned diffusion models across different taxonomic levels — starting from broad classifications such as Class and Order , refining through Family and Genus , and ultimately distinguishing at the Species level. This hierarchical learning strategy first captures coarse-grained morphological traits shared by species with common ancestors, facilitating knowledge transfer, before refining fine-grained differences for species-level distinction. As a result, TaxaDiffusion enables accurate generation even with limited training samples per species. Extensive experiments on three fine-grained animal datasets demonstrate that TaxaDiffusion outperforms existing approaches, achieving superior fidelity in fine-grained animal image generation. Project page: https://amink8.github.io/TaxaDiffusion/ 1 Introduction Figure 1 : Taxonomy encodes a rich hierarchical structure for categorizing life. We propose TaxaDiffusion to leverage such knowledge to enable fine-grained, controllable image generation. Compared to Zero-Shot generation with vanilla Stable Diffusion [ 39 ] and LoRA fine-tuning [ 16 ] , TaxaDiffusion achieves higher accuracy and captures fine details that align closely with real images. Figure 2 : Generative examples of our approach on the FishNet dataset [ 19 ] . As we progress through the taxonomy tree from Class to Order , Family , Genus , and finally Species , our model refines its understanding of distinguishing traits, generating realistic images that capture the unique visual characteristics at each level. For rare species with limited training samples, such as “Amphichaetodon Howensis” ( 4 samples), “Amphichaetodon Melbae” ( 1 sample), and “Chaetodon Humeralis” ( 5 samples), our taxonomy-informed, progressive training approach enables effective knowledge transfer from related species, allowing the model to generate morphologically accurate species images even with sparse data. The corresponding ground-truth images from FishNet are shown on the right. Diffusion models have revolutionized the field of image generation [ 35 , 14 , 6 ] . By progressively denoising sampled noise, these models can generate diverse, high-quality images with fine details. Further enhanced by vision-language models [ 36 ] and various control mechanisms [ 28 , 3 , 60 , 57 ] , diffusion models can produce complex compositions of common concepts (e.g., objects and attributes) while faithfully aligning with user inputs. However, when it comes to generating fine-grained concepts, specifically animal species , diffusion models still face significant challenges. Given a species name, existing models often fail to generate morphologically accurate images or images that precisely reflect the identity of species. Two key domain characteristics contribute to this issue. First, animals are highly dynamic objects with considerable degrees of freedom. They exhibit various poses and movements, resulting in significant intra-class variation. Second, with millions of animal species on Earth, collecting sufficient samples to capture both intra-class variations and inter-class distinctions is a daunting task. These challenges paint a grim picture for accurate animal generation. We ask: How can we learn high-quality generative models for animals without a large amount of data per species? Our key insight is that, despite the vast number of animal species, they are not independent categories. Instead, it took half a billion years for the Animalia Kingdom to evolve from a few species to millions, and species that diverged more recently often share similar visual traits. While CLIP [ 36 , 27 ] , commonly used for prompting in diffusion models, may have learned species relationships from image-text pairs, it primarily relies on correlations. These correlations are only reliable when sufficient data is available — often not the case for fine-grained animal species. Therefore, we propose leveraging Taxonomy [ 30 ] , a classification system that has been continuously developed by biologists over decades, if not centuries, to categorize species (Figure 1 ). A typical taxonomy consists of 7 hierarchical levels [ 7 , 15 ] : Kingdom , Phylum , Class , Order , Family , Genus , and Species . Species sharing a common ancestor often exhibit similar characteristics, such as body parts, layouts, and valid poses. The closer two species are on the taxonomy tree, the more similar their appearance tends to be. For species within the same Genus or Family, their distinctions often lie in subtle variations in shape, color, or pattern. To exploit these relationships, we ask: How can we use taxonomy in training diffusion models? One straightforward approach is to replace the common name of a species (e.g, “cat”) with its full taxonomic hierarchy (e.g., “Animalia”-“Chordata”-“Mammalia”-“Carnivora”-“Felidae”-“Felis”-“Catus”) [ 45 , 20 ] . However, introducing all taxonomic levels at once overwhelms the training process, making it hard for the model to effectively learn taxonomic relationships. To address this issue, we propose a progressive training approach that teaches the model taxonomy step by step, from root to leaf. We hypothesize that this progressive strategy will encourage the model to first learn common body structures, layouts, and poses shared across species, and then refine its focus on subtle differences in shape, color, and pattern — thus unlocking the full potential of a generative model for synthesizing high-quality animal images. Figure 2 presents actual examples generated by our method. We implement our approach, TaxaDiffusion , as follows, using an animal image dataset labeled with full taxonomic names. Building upon a pre-trained diffusion model (e.g., Stable Diffusion v1-5 [ 39 ] ), we retrain a new condition encoder to process taxonomy information (see Section 3 for details). Instead of feeding the full taxonomic name for each image at once, we break the training into multiple progressive stages, each corresponding to a different taxonomy level. At the i -th stage, we input the taxonomic name up to the i -th level, encouraging the model to learn common visual properties from images labeled with the same taxonomic name up to that level. For example, “lion” and “hyena”, rivals in many African ecosystems, will be treated as belonging to the same category up to the Order level, with both labeled as “Animalia”-“Chordata”-“Mammalia”-“Carnivora.” As a result, the diffusion model can explicitly learn the common visual characteristics of “Carnivora” from images of many species within this group. By progressively refining visual granularity, TaxaDiffusion further enables species with limited data to leverage information from related species, promoting knowledge transfer. Specifically, for a rare species, the model only needs to learn how it differs from other species within the same Genus at the final stage — usually subtle variations in shape, color, and pattern. Its other visual characteristics can be learned from images of related species in earlier stages. We empirically validate TaxaDiffusion for fine-grained species generation across three challenging datasets: FishNet [ 19 ] , BIOSCAN-1M [ 9 ] , and iNaturalist [ 48 ] , which cover a diverse range of species. Specifically, FishNet [ 19 ] includes around

Read the complete text