Chirpy3D: Part-Aware Multi-View Diffusion for Creative Fine-Grained Object Generation

Kam Woh Ng, Jing Yang, Jia Wei Sii, Chee Seng Chan, Jiankang Deng, Yi-Zhe Song, Tao Xiang, Xiatian Zhu · 2025

Read Chirpy3D: Part-Aware Multi-View Diffusion for Creative Fine-Grained Object Generation by Kam Woh Ng, Jing Yang, Jia Wei Sii, Chee Seng Chan, Jiankang Deng, Yi-Zhe Song, Tao Xiang, Xiatian Zhu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-04144.txt.

Introduction

Kam Woh Ng and co-authors build a 3D generator that understands bird parts in this 2025 paper. Fine detail is the challenge: each species has its own beak, wings, and tail. Their system, Chirpy3D, learns a layered space of parts from flat photos alone. No 3D training data, camera angles, or hand labels are needed. Off-the-shelf part masks from 2D images give the spatial guidance. A self-taught consistency rule keeps views in structural agreement.

The learned space lets users swap, blend, and combine parts freely. New hybrids, even unseen part mixes, come out coherent. Multi-view diffusion is the engine that paints the object from many angles. The work targets creative generation with fine control. It uses birds as the test case for part-level structure. Other finely divided object families could follow the same path.

Catalog record

CREATORKam Woh Ng, Jing Yang, Jia Wei Sii, Chee Seng Chan, Jiankang Deng, Yi-Zhe Song, Tao Xiang, Xiatian Zhu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-04144

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Chirpy3D: Part-Aware Multi-View Diffusion for Creative Fine-Grained Object Generation Kam Woh Ng, Jing Yang, Jia Wei Sii, Chee Seng Chan, Jiankang Deng, Yi-Zhe Song, Tao Xiang, Xiatian Zhu Abstract Understanding and generating the fine-grained structure of objects – such as birds with species-specific beaks, wings, and tails – is a long-standing challenge in computer vision. We propose Chirpy3D, a part-aware multi-view diffusion framework that learns a hierarchical part latent space from unposed 2D images, using only off-the-shelf 2D part segmentation masks as spatial guidance – without requiring any 3D data, camera poses, or manual part annotations. This latent space enables intuitive part-level swapping, interpolation, and zero-shot composition. A self-supervised feature consistency loss further encourages structural alignment across views, allowing coherent generation even with hybrid or unseen part combinations. Our core contribution is the controllable part-aware latent space and multi-view diffusion model. Downstream 3D generation is supported via any differentiable renderer such as NeRF but is orthogonal to the main framework, making Chirpy3D a flexible foundation for creative object generation in the absence of structured 3D data. Figure 1 : Creative 3D creatures generated by Chirpy3D . The creativity is achieved through part-level sampling in a learned latent space. The selected parts are composed into a textual prompt, which is then fed into a multi-view diffusion model to guide the 3D representation optimization (NeRF [ 46 ] ) via SDS loss [ 52 ] . 1 Introduction Understanding and modeling the fine-grained structure of objects is a long-standing challenge in computer vision and graphics [ 2 , 15 , 35 , 25 , 75 , 51 , 47 , 48 , 22 ] . Many real-world objects – such as animals [ 23 ] , vehicles [ 36 ] , and furniture [ 4 ] – share global anatomical structures while exhibiting rich part-level variation. For example, birds [ 65 ] may differ in beak shapes, wing patterns, or tail geometries depending on species. Learning such part-aware representations opens up exciting opportunities for controllable image and 3D object generation (see Figure 1 ). Diffusion models have recently driven major progress in both 2D [ 56 , 58 , 13 ] and 3D generation [ 17 , 67 , 70 ] . In particular, multi-view diffusion models [ 44 , 59 , 43 , 42 , 37 , 74 , 33 , 31 , 60 ] have demonstrated the ability to synthesize 3D-consistent images from a single view. More recent approaches such as PartGen [ 6 ] go further by introducing explicit part-level supervision for multi-view 3D generation via segmentation and completion pipelines. However, most existing methods are trained on curated 3D datasets [ 10 , 68 ] and focus on general-purpose generation. They lack the flexibility for part-wise editing, creative recombination, or structural control through latent space. In contrast, learning part-aware representations for fine-grained categories (such as novel birds with hybrid, species-specific traits) remains underexplored, particularly in settings where 3D annotations or posed multi-view images are unavailable. PartCraft [ 48 ] pioneered 2D part-aware generation by learning discrete per-class textual embeddings for each part, but these do not form a continuous latent space – preventing smooth interpolation, random sampling, or generalization to unseen combinations, and it operates only in 2D. We extend this by learning a continuous part latent space with a Gaussian prior, enabling structured interpolation and sampling, integrated with a multi-view diffusion backbone for downstream 3D generation. In this paper, we focus on learning a part-aware, multi-view diffusion model trained on unposed, unlabeled 2D images from fine-grained object categories, without 3D ground truth, camera poses, or keypoint annotations. To guide part-attention learning, we use 2D part segmentation masks from an off-the-shelf detector [ 48 ] as spatial guidance. Our key idea is to learn a hierarchical part latent space enabling compositional control over object parts, supporting operations such as interpolation, swapping, and zero-shot composition. We further introduce a self-supervised feature consistency loss that encourages structural alignment across views, helping generate coherent shapes even with hybrid or unseen parts. While we demonstrate 3D generation via NeRF [ 46 ] and 3DGS [ 32 ] , the choice of 3D backend is orthogonal – our core contribution is the controllable part-aware latent space. Figure 2 : Overview of Chirpy3D . Chirpy3D takes (a) a set of unposed 2D images from multiple fine-grained species (e.g., birds) and (b) learns to decompose each object into a set of underlying parts (e.g., head, wings, torso, legs, tail) within a hierarchical part latent space – species embedding \mathbf{s} captures global species characteristics while part-level embedding \mathbf{p} captures fine-grained part variations. (c-d) The part latent space is regularized with a Gaussian prior for smooth interpolation and novel part sampling, with shared part-specific position embeddings ( PE ) enabling cross-species alignment. (e) Part embeddings are projected via a learnable function g into part-aware textual embeddings to condition the multi-view diffusion model ( e.g . , MVDream [ 60 ] ) to generate multi-view images. (f) A self-supervised feature consistency loss is applied to enforce structural and semantic coherence across views, improving the realism and alignment of both seen and unseen parts. (g) During inference, Chirpy3D supports both reconstruction and creative generation – either directly using learned part latent codes or sampling/interpolating within the part latent space – which guides 3D representation learning ( e.g . , NeRF or 3DGS) via SDS optimization. 2 Related work Fine-grained generation. Research on fine-grained visual understanding has primarily focused on 2D tasks, including fine-grained recognition [ 76 , 5 , 12 , 66 , 47 ] and 2D object generation [ 62 , 72 , 40 , 11 , 39 , 7 ] . In contrast, fine-grained 3D generation remains underexplored, with most existing works focusing on relatively simple, rigid categories such as chairs and cars [ 73 , 17 , 61 , 29 , 19 , 20 , 50 ] . Deformable fine-grained categories, such as birds and dogs, pose unique challenges due to complex textures, diverse part configurations, and high pose variability, while obtaining 3D data remains expensive. Our work pushes beyond these limitations to enable the creative synthesis of novel, fine-grained objects directly from unposed 2D images. Diffusion models. Diffusion models have recently been applied to multi-view generation, though they often suffer from multi-view inconsistencies — commonly known as the “Janus problem.” Text-to-multi-view methods such as MVDream [ 60 ] and SPaD [ 31 ] generate object views from text prompts, while methods like Zero123 [ 44 ] , Zero123++ [ 59 ] , and EscherNet [ 33 ] infer novel views from a single image. These methods focus on rendering consistent views of known objects but lack the ability to model fine-grained part-level variations or creative generation of novel objects. Our approach extends this line of work by enabling the generation of entirely new fine-grained objects in a zero-shot setting, including novel part combinations and species-level diversity. Part-aware object generation. Part-aware generation has been extensively explored in 2D, where methods [ 1 , 48 ] such as PartCraft [ 48 ] leverage attention-based objectives to disentangle and recombine object parts. However, these methods typically operate on predefined or salient parts of specific object categories and do not

Read the complete text