TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life

Mridul Khurana Amin Karimi Monsefi Justin Lee Medha Sawhney David Carlyn Julia Chae Jianyang Gu Rajiv Ramnath Sara Beery Wei-Lun Chao Anuj Karpatne Cheng Zhang · 2026

Read TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life by Mridul Khurana Amin Karimi Monsefi Justin Lee Medha Sawhney David Carlyn Julia Chae Jianyang Gu Rajiv Ramnath Sara Beery Wei-Lun Chao Anuj Karpatne Cheng Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26128.txt.

Introduction

Khurana and co-authors present TaxaAdapter, a method for generating faithful images of species. Over ten million species exist, and many differ by tiny visual traits. Text-to-image models often miss those traits even in sharp outputs. TaxaAdapter feeds embeddings from a vision taxonomy model into a frozen diffusion model. Text control over pose, style, and background stays intact. The method lifts form fidelity and species accuracy over strong baselines, the authors report. It even handles species with few or no training images.

Catalog record

CREATORMridul Khurana Amin Karimi Monsefi Justin Lee Medha Sawhney David Carlyn Julia Chae Jianyang Gu Rajiv Ramnath Sara Beery Wei-Lun Chao Anuj Karpatne Cheng Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61684
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TaxaAdapter: Vision Taxonomy Models are Key to Fine-grained Image Generation over the Tree of Life Mridul Khurana Amin Karimi Monsefi Justin Lee Medha Sawhney David Carlyn Julia Chae Jianyang Gu Rajiv Ramnath Sara Beery Wei-Lun Chao Anuj Karpatne Cheng Zhang Abstract Accurately generating images across the Tree of Life is difficult: there are over 10M distinct species on Earth, many of which differ only by subtle visual traits. Despite the remarkable progress in text-to-image synthesis, existing models often fail to capture the fine-grained visual cues that define species identity, even when their outputs appear photo-realistic. To this end, we propose TaxaAdapter, a simple and lightweight approach that incorporates Vision Taxonomy Models (VTMs) such as BioCLIP to guide fine-grained species generation. Our method injects VTM embeddings into a frozen text-to-image diffusion model, improving species-level fidelity while preserving flexible text control over attributes such as pose, style, and background. Extensive experiments demonstrate that TaxaAdapter consistently improves morphology fidelity and species-identity accuracy over strong baselines, with a cleaner architecture and training recipe. To better evaluate these improvements, we also introduce a multimodal Large Language Model-based metric that summarizes trait-level descriptions from generated and real images, providing a more interpretable measure of morphological consistency. Beyond this, we observe that TaxaAdapter exhibits strong generalization capabilities, enabling species synthesis in challenging regimes such as few-shot species with only a handful of training images and even species unseen during training. Overall, our results highlight that VTMs are a key ingredient for scalable, fine-grained species generation. † † footnotetext: † [email protected], [email protected], [email protected] Models and code are available at imageomics.github.io/TaxaAdapter Figure 1 : Existing text-to-image models struggle with fine-grained species details. We propose TaxaAdapter to inject VTM-derived embeddings into a frozen diffusion model to make fine-grained synthesis taxonomy-aware, enabling: (a) fine-grained species synthesis with accurate morphological details, (b) diverse and high-fidelity trait synthesis, (c) free-form text control, and (d) strong out-of-distribution generalization. 1 Introduction Generating images of fine-grained species across the Tree of Life is a problem of growing importance in biodiversity science that presents a unique challenge for text-to-image synthesis. Unlike broad object categories, species identity is often defined by subtle morphological cues [ 39 ] such as slight differences in beak curvature or plumage patterns. For example, closely related bird species may differ only in faint head or throat markings (Figure 1 -(a)). In addition to this challenge, the underlying distribution of biodiversity images is extremely long-tailed: among the more than 10 M species on Earth, many have only a handful of curated photos. This motivates new methods for fine-grained generation that enables biologists to generate faithful visualizations of target species even with little to no data, under varying contexts (e.g., pose) and rendering styles (e.g., schematic sketches) to discover diagnostic traits of closely related taxa. Recent advances in text-to-image diffusion models [ 32 , 28 , 2 , 17 , 1 , 48 ] trained on large-scale web data have reshaped image synthesis by strengthening the alignment between textual descriptions and visual details. However, these models predominantly learn representations organized around coarse semantic categories (e.g., “bird” or “fish”) rather than precise taxonomic distinctions (e.g., Merops bullockoides ). Hence, they often struggle to capture the subtle morphological cues of individual species, producing images that appear realistic but belong to the wrong species (see example images of FLUX.1-dev [ 17 ] in Figure 1 -(a)). A natural source of knowledge for guiding species generation is the taxonomic hierarchy of species that organizes species into related groups and enables information sharing across related taxa. There is a growing body of work on using hierarchical knowledge in diffusion models to improve fine-grained generation [ 23 , 14 , 27 ] . While promising, these approaches typically require training taxonomic embeddings from scratch, making them computationally expensive to scale and limiting their ability to generalize to species not observed during training, hindering out-of-distribution generation performance. There have also been remarkable advances in building Vision Taxonomy Models (VTMs) such as BioCLIP [ 37 , 5 ] , TaxaBind [ 35 ] , and BioTrove-CLIP [ 44 ] that learn embeddings aligned with both vision and taxonomy at scale. We hypothesize that incorporating such embeddings into generative models can inject taxonomy-aligned visual knowledge necessary for fine-grained generation. However, existing VTMs are primarily designed for recognition tasks [ 37 , 5 , 35 , 44 ] , and their taxonomy-aware representations have not yet been tied to generative objectives. This raises a key question: How can we unify the taxonomy–image alignment knowledge of VTMs with the power of generative models? To this end, we propose TaxaAdapter , a lightweight plug-in that injects image-aligned taxonomy embeddings from VTMs into a frozen text-to-image diffusion model. Our method employs a novel dual conditioning mechanism that allows the generative model to leverage both free-form textual prompts and visual-taxonomy representations. By aligning the generative process with the visual–taxonomic knowledge encoded in VTMs, TaxaAdapter enables more accurate fine-grained species synthesis (Figure 1 -(b)) while preserving flexible attribute control through text prompts (Figure 1 -(c)). Extensive experiments show that TaxaAdapter significantly improves species-level fidelity and morphological consistency compared with strong baselines. To better evaluate these improvements, we introduce a novel Multimodal Large Language Model (MLLM)-based trait evaluation framework that summarizes trait-level descriptions from generated and real images, providing an interpretable measure of morphological fidelity. Beyond this, we show that TaxaAdapter exhibits strong generalization capabilities, enabling synthesis for rare species with limited training images as well as species never seen during training. To the best of our knowledge, TaxaAdapter is the first framework that bridges diffusion models and pre-trained VTMs, closing the gap between taxonomic knowledge and visual similarity for generative goals and offering a practical route toward scalable, fine-grained species generation across the Tree of Life. Remark. Generating species images is not a new topic. There are many prior works; many focus on model architectures and learning recipes. However, none really scale to a large number of species. We argue that overcoming the long-tailed and fine-grained nature of this problem requires incorporating taxonomy knowledge over the entire Tree of Life, and accordingly leverage embeddings produced by recent advances in VTMs. This enables us to achieve much stronger generation quality than baselines while maintaining a simple framework that is easy to extend. In our humble opinion, such simplicity — without complicated architectures and training loss invention — is the key strength of our work. 2 Related Work Fine-grained Generation aims to model categories with subtle visual differences, such as closely related fish species, fine facial attributes, or detailed object variations. Prior work explores several directions. Sketch-based control methods guide synthesis through user-provided structure, enabling local detail refinement [ 24 , 53 , 18 , 48 ] . Attribute-manipulation approaches condition on latent or semantic attributes to adjust specific features [ 11 , 15 , 46 , 6 , 22 , 47 ] . Personalized generation aims to preserve identity or instance-specific features by tuning part of the model [ 51 , 33 , 19 , 42 , 34 ] . Recently, TaxaDiffusion [ 23 ] and PhyloDiffusion [ 14 ] explore conditioning on hierarchical knowledge (e.g., taxonomy) for species image synthesis. These models train hierarchical embeddings in the conditioning space of

Read the complete text