Representing 3D Shapes with 64 Latent Vectors for 3D Diffusion Models

In Cho, Youngbeom Yoo, Subin Jeon, Seon Joo Kim · 2025

Read Representing 3D Shapes with 64 Latent Vectors for 3D Diffusion Models by In Cho, Youngbeom Yoo, Subin Jeon, Seon Joo Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08737.txt.

Introduction

In Cho and three co-authors compress 3D shapes for diffusion models in twenty twenty-five. Efficient 3D diffusion needs a compact latent space, usually built by a variational autoencoder. Their encoder, COD-VAE, squeezes a shape into just sixty-four latent vectors. Quality is meant to survive that squeeze.

A two-stage autoencoder scheme drives the compression. The encoder progressively condenses point clouds through intermediate patches. The decoder rebuilds dense triplanes from the latent vectors rather than decoding neural fields directly. That swap cuts heavy computation. Uncertainty-guided token pruning skips work in simple regions, saving more. Experiments showed sixteenfold compression against the baseline while maintaining quality. Readers get 3D generation that fits in a far smaller latent space.

Catalog record

CREATORIn Cho, Youngbeom Yoo, Subin Jeon, Seon Joo Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-08737

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Representing 3D Shapes with 64 Latent Vectors for 3D Diffusion Models In Cho, Youngbeom Yoo, Subin Jeon, Seon Joo Kim Abstract Constructing a compressed latent space through a variational autoencoder (VAE) is the key for efficient 3D diffusion models. This paper introduces COD-VAE that encodes 3D shapes into a COmpact set of 1D latent vectors without sacrificing quality. COD-VAE introduces a two-stage autoencoder scheme to improve compression and decoding efficiency. First, our encoder block progressively compresses point clouds into compact latent vectors via intermediate point patches. Second, our triplane-based decoder reconstructs dense triplanes from latent vectors instead of directly decoding neural fields, significantly reducing computational overhead of neural fields decoding. Finally, we propose uncertainty-guided token pruning, which allocates resources adaptively by skipping computations in simpler regions and improves the decoder efficiency. Experimental results demonstrate that COD-VAE achieves 16 × \times compression compared to the baseline while maintaining quality. This enables 20.8 × 20.8\times speedup in generation, highlighting that a large number of latent vectors is not a prerequisite for high-quality reconstruction and generation. The code is available at https://github.com/join16/COD-VAE . 1 Introduction Learning a compressed latent space through a variational autoencoder [ 22 ] (VAE) has become a crucial component of recent diffusion models. Due to their iterative inference procedures, generating contents directly in the observation space ( e.g. , pixels) is computationally intractable. To address this challenge, most state-of-the-art generative models adopt a latent diffusion framework: a VAE first compresses the input data into a lower-dimensional latent space, and diffusion models are trained within this latent space. Constructing a compact, well-structured latent space is even more crucial in the 3D domain. 3D objects are inherently irregular, sparse and continuous, which makes direct generation in the 3D domain more challenging. Previous research has explored various explicit 3D representations, e.g. , points [ 47 ] , voxels [ 40 ] , and octrees [ 49 ] . These explicit representations typically demand larger latent sizes and specifically designed networks to process, making the training and the scaling of generative models difficult. Figure 1 : (top) Reconstruction IoU (left) and Rendering-FID of the generation results (right) with varying numbers of latent vectors ( M ). Our COD-VAE outperforms VecSet using 16 \times fewer latent vectors, achieving 20.8\times generation speedup. (bottom) VecSet with M=64 struggles to capture details, while our model accurately reconstructs detailed and complex shapes of the objects. Recently, 3DShape2VecSet [ 56 ] (VecSet) established a foundational framework for encoding 3D shapes into a set of 1D latent vectors. These 1D vectors provide a more compact latent space than the explicit representations, and are highly compatible with modern transformer implementations ( e.g. , FlashAttention [ 10 ] ). Due to their simplicity and effectiveness, 1D latent vectors have been actively extended in subsequent research [ 58 , 36 , 59 , 3 , 12 , 54 , 7 , 60 ] . The core component of VecSet is the cross-attention layer, which directly projects an input point cloud into 1D latent vectors and maps them back to continuous neural fields. These direct mappings, operating as learnable downsampling and interpolation, leads to the latent vectors only with a moderate compression ratio. Due to the limited compression ratio, VecSet still requires sufficiently many latent vectors for high-quality reconstruction, resulting in excessive computational costs of 3D generative models. Moreover, directly mapping latent vectors to neural fields through cross-attention creates another computational bottleneck. In this paper, we introduce COD-VAE, a VAE that encodes 3D shapes into a CO mpact set of 1 D vectors with an improved compression ratio. COD-VAE replaces the direct mappings between points and latent vectors with a two-stage autoencoder scheme. This scheme enables our model to construct a significantly compressed latent space, thereby accelerating subsequent diffusion models. It achieves high-quality reconstruction with 16\times fewer latent vectors, as well as efficiency improvement in neural fields decoding. The two-stage autoencoder scheme leverages intermediate representation spaces with a moderate compression ratio. These intermediate representations serve as bridges between 3D points and latent vectors. Concretely, our COD-VAE incorporates the following core components: • Our encoder introduces intermediate point patches, which are obtained by leveraging the attention-based downsampling of VecSet as a learnable point patchifier. Each encoder block progressively transforms high-resolution points into compact latent vectors by first mapping them into intermediate point patches and then further compressing them into latent vectors. The global information of the latent vectors are mapped back to points at the end of the block, further refining high-resolution features. • Our decoder leverages triplanes [ 4 , 13 ] as intermediate representations for efficient yet effective decoding process. Inspired by [ 52 ] , we treat dense triplane embeddings as mask tokens, and reconstruct them from the latent vectors using transformers. This design significantly reduces computational overheads of the neural fields decoding while also improving the reconstruction quality. • Apart from the efficient neural fields decoding, processing dense triplanes through transformers requires excessive computational resources. To mitigate this, we propose uncertainty-guided token pruning that reduces computations in simple regions, thereby achieving further efficiency improvement. Our decoder employs an auxiliary uncertainty head at the beginning, trained to predict reconstruction errors. By pruning regions with lower uncertainty, our decoder prioritizes computational resources for reconstructing more complex regions. The compact latent space constructed by COD-VAE enables efficient and high-quality 3D generation. Experimental results on ShapeNet [ 5 ] , and Objaverse [ 11 ] verify the effectiveness of our method, achieving 20.8\times speedup in generation without sacrificing performance (see Figure 1 ). These results highlight that a large number of latent vectors is not a prerequisite for high-quality reconstruction and generation. We only need 64 latent vectors to surpass the results of VecSet with 512, or even 1024 latent vectors. 2 Related work 3D generative models. Previous methods for 3D generation can be categorized into two main approaches. The first approach employs an optimization-based pipeline with 2D diffusion priors to generate 3D objects [ 37 , 28 , 26 , 42 , 46 , 39 ] . These methods incur extensive computational costs due to the optimization process per object. Conversely, the second approach directly trains generative models on 3D datasets, which have shown promising results with significantly faster inference times. These native 3D generative models have been widely explored, including GAN-based models [ 1 , 44 , 17 , 4 , 14 , 35 ] , autoregressive models [ 45 , 33 , 30 , 50 ] , and normalizing flows [ 51 , 23 , 21 ] . Recent breakthroughs of diffusion models have also inspired a wave of research in the 3D domain [ 34 , 62 , 43 , 25 , 19 , 31 ] , which typically adopt the latent diffusion framework [ 32 ] . 3D VAEs. Several studies have explored 3D VAEs with various latent representations. One line of works employs dense structures to model the latent space [ 9 , 24 , 27 ] , which lead to quadratic [ 24 , 27 ] or cubic [ 9 ] computational costs as the resolution increases. Other works have designed a more efficient latent space by leveraging sparse structures, such as points [ 47 ] , octrees

Read the complete text