Enhanced Mixture 3D CGAN for Completion and Generation of 3D Objects Thanks: This paper was produced by the IEEE Publication Technology Group. They are in Piscataway, NJ. Thanks: Manuscript received April 19, 2025; revised August 16, 2021. Yahia Hamdi, Nicolas Andrialovanirina Abstract The generation and completion of 3D objects represent a transformative challenge in computer vision. Generative Adversarial Networks (GANs) have recently demonstrated strong potential in synthesizing realistic visual data. However, they often struggle to capture complex and diverse data distributions, particularly in scenarios involving incomplete inputs or significant missing regions. These challenges arise mainly from the high computational requirements and the difficulty of modeling heterogeneous and structurally intricate data, which restrict their applicability in real-world settings. Mixture of Experts (MoE) models have emerged as a promising solution to these limitations. By dynamically selecting and activating the most relevant expert sub-networks for a given input, MoEs improve both performance and efficiency. In this paper, we investigate the integration of Deep 3D Convolutional GANs (CGANs) with a MoE framework to generate high-quality 3D models and reconstruct incomplete or damaged objects. The proposed architecture incorporates multiple generators, each specialized to capture distinct modalities within the dataset. Furthermore, an auxiliary loss-free dynamic capacity constraint (DCC) mechanism is introduced to guide the selection of categorical generators, ensuring a balance between specialization, training stability, and computational efficiency, which is critical for 3D voxel processing. We evaluated the model’s ability to generate and complete shapes with missing regions of varying sizes and compared its performance with state-of-the-art approaches. Both quantitative and qualitative results confirm the effectiveness of the proposed MoE-DCGAN in handling complex 3D data. Fig. 1: Our MoE-CGAN generates 3D shapes. The 3D objects shown were reconstructed using the Marching Cubes. I Introduction In recent years, the rise of data-driven approaches, particularly deep learning methods, has led to an increased demand for high-quality datasets encompassing training, validation, and testing splits. These datasets are critical for developing and evaluating robust machine learning models. However, in many domains, such as marine biology, large-scale data collection remains challenging or infeasible. This limitation is especially pronounced in the study of rare biological phenomena, where data availability is inherently scarce. Furthermore, the use of real-world data often raises privacy and ethical concerns, further constraining access to sufficiently large and diverse datasets for effective model development. The scarcity of high-quality labeled data is a widespread challenge that affects not only the biological field but also domains such as object recognition and the study of porous media [ 1 , 2 ] , where data collection is often expensive and time-consuming. In the context of 3D objects classification, this issue is particularly pronounced. The task remains difficult largely due to the limited availability of large, comprehensive datasets. In some cases of study, only datasets containing damaged or incomplete objects are accessible. This presents a significant problem, especially when using spherical Fourier descriptors for 3D analysis and classification, as these methods typically require complete and intact shapes for accurate computation. To overcome this limitation, it is necessary to develop methods capable of efficiently exploiting all available data, even if incomplete, to reconstruct or approximate the original shape of the objects. This then enables the reliable calculation of spherical Fourier descriptors and the accurate classification of objects, despite their fragmented state. A variety of advanced methods have been introduced for high-quality image generation, including ProgressiveGAN, SplittingGAN, and Wasserstein GAN with Gradient Penalty (WGAN-GP), each contributing to improvements in training stability and visual realism. These generative models have proven particularly valuable in the field of 3D object completion and reconstruction, where the challenge lies in restoring missing or damaged parts of objects. Approaches such as symmetry-based modeling [ 3 , 4 ] and deep learning methods including GANs [ 5 ] and autoencoders [ 6 ] have demonstrated strong capabilities in predicting and filling in incomplete shapes using partial input data. Furthermore, recent innovations like attention mechanisms [ 7 , 8 ] and multi-scale feature extraction have significantly enhanced the performance of these models, enabling more accurate and context-aware reconstructions in complex scenarios. In parallel, diffusion models have emerged as a promising alternative for 3D data generation, offering practical benefits and strong potential for real-world deployment [ 9 ] . While diffusion models are known for their high-quality generation capabilities, GANs still produce competitive results. However, the slower sampling speed of diffusion models leads to longer training times. Moreover, [ 10 ] found that diffusion models can leak twice as much private information as GANs, raising significant privacy concerns. Given these factors, we choose to focus on improving GANs, viewing them as a more balanced and practical solution. The concept of combining multiple neural networks through a gating mechanism was originally introduced to improve performance in multi-speaker phoneme recognition tasks [ 12 ] . In this early formulation, a collaborative loss function encouraged networks to contribute complementary outputs. Subsequent advancements refined this framework by introducing a competitive objective function, which encouraged each network to specialize in distinct subspaces of the problem [ 13 ] . This approach evolved into the well-established Mixture of Experts (MoE) architecture, which has since been widely adopted across various machine learning domains. MoE has gained increasing prominence, particularly in large-scale neural architectures such as Large Language Models (LLMs), where it has contributed to state-of-the-art performance in models like DeepSeek-V3. Major research organizations including Microsoft, OpenAI, and Google have actively advanced MoE-based systems due to their flexibility and ability to improve model capacity without proportional increases in computational cost. Inspired by this paradigm, we propose a novel MOE-CGAN architecture that incorporates a MoE mechanism to address key challenges in otolith shape completion and generation; an essential task in biological and ecological research. Otoliths exhibit complex morphological variations that encode information related to species identity, environmental conditions, and growth dynamics. This work extends a prior 3D CGAN-based framework by enhancing its capacity to model such variability. Conventional single-generator GANs often struggle to capture the full diversity of otolith structures, resulting in mode collapse or overly simplified outputs. In contrast, our architecture employs multiple specialized generators, coordinated through a gating network that adaptively assigns input data (e.g., partial or noisy otolith shapes) to the most appropriate expert. This design not only enhances the quality and realism of shape completion but also enables the generation of biologically potential variations, thereby supporting downstream applications such as species classification, ecological monitoring, and fisheries resources management. The key contributions of our work can be summarized as follows : • We propose a novel model, called MOE-CGAN, to effectively address common challenges in generative modeling such as mode collapse and training instability particularly in the context of otolith shape completion and generation. This approach introduces a new architectural paradigm that combines the strengths of MoE with CGANs. • We introduce geometry-aware specialization by employing multiple 3D CGAN, each trained to focus on distinct geometric characteristics using optimal auxiliary-loss-free based DCC load balancing strategy. • We conduct comprehensive experiments across multiple datasets and evaluation metrics, demonstrating that our method outperforms some existing state-of-the-art approaches in both shape fidelity and morphological diversity. The remainder of this paper is organized as follows. Section 2 provides a review of generation and completion-based models. Section 3 describes the overall model architecture, including the MoE pipeline and training strategy. Performance results and evaluations are presented in Section 4, and Section 5 concludes the paper with directions for future work. II Related Works Numerous research efforts have focused on improving the stability of GAN training and enhancing the quality of generated and completed outputs. Introduced by [ 14 ] and [ 15 ] , GANs have emerged as a powerful alternative to traditional generative models such as autoencoders and variational autoencoders, significantly advancing the realism of synthesized images. Since their introduction, GANs have evolved rapidly, with landmark contributions such as [ 16 ] , which demonstrated the generation of highly photorealistic human faces. These advancements have established GANs as a foundational approach in deep learning–based synthetic data generation, offering superior realism and diversity compared to earlier techniques. Meanwhile, diffusion models have recently gained significant attention as an alternative generative framework, delivering state-of-the-art performance in image synthesis tasks [ 9 ] , [ 10 ] , [ 11 ] . Although, [ 17 ] provide a survey that covers approaches leveraging diffusion models for manipulating 3D content represented in both explicit and implicit forms. The survey highlights techniques for per-scene optimization in 3D generation and editing, methods for novel view synthesis guided by diffusion, as well as strategies for learning 3D data distributions from existing datasets using diffusion-based frameworks. While diffusion models provide improved stability and superior sample quality, they fundamentally differ from GANs in terms of architecture and training dynamics. Importantly, these two paradigms are not mutually exclusive; they can complement one another depending on application needs. In the domain of 3D shape generation, two primary approaches are commonly adopted: part-conditioned and unconditioned generation. Part-conditioned methods rely on datasets annotated with structural parts. These approaches often use variational autoencoders (VAEs) to encode each individual part into its own latent distribution [ 18 ] , enabling the synthesis of novel shapes by sampling and assembling parts from their respective latent spaces. Conversely, unconditioned methods generate entire 3D shapes directly from random latent vectors, without requiring part annotations or predefined structural constraints. This approach provides greater generative flexibility and is particularly advantageous when annotated data is scarce. An unsupervised framework known as pcl2pcl was proposed in [ 19 ] , utilizing unpaired data comprising complete shapes from synthetic models and partial scans from real-world sources. This method employs two separate autoencoders: one for complete shapes and another for partial inputs. A latent space mapping is learned to bridge the partial and complete shape domains. Additionally, [ 20 ] introduced a GAN inversion-based unsupervised method for 3D shape completion. This approach refines the latent code of a pre-trained shape-generating GAN to produce complete shapes aligned with partial observations, achieving performance on par with supervised techniques. Another work proposed by [ 21 ] involves learning probabilistic distributions over 3D shapes directly from unlabeled 2D views, without the need for explicit 3D supervision or viewpoint annotations. By integrating a differentiable projection module, PrGAN enables unsupervised estimation of both 3D shape and viewpoint, while also supporting novel view synthesis from a single image. Furthermore, SP-GAN, proposed by [ 22 ] , is an unsupervised generative framework for synthesizing 3D shapes in the form of point clouds. Unlike prior methods, SP-GAN does not require part annotations and enables part-aware control by leveraging a global spatial prior derived from uniformly sampled spherical points. It combines this with local latent codes to enrich fine-grained geometric details, resulting in high-quality and diverse shape generation. More recently, a 3D cascade GAN model based on an encoder-decoder architecture was proposed by [ 23 ] to reconstruct shapes from single-view depth images. This model aims to select important codes, capture non-local relationships in latent 3D space, and incorporate a self-attention layer to stabilize model learning and refine the reconstructed shapes. Despite their strong potential, GAN-based approaches face some limitations when applied to 3D object generation and completion. A major challenge is mode collapse, where the model fails to capture the full variability of complex 3D data. Training instability further complicates the process, as adversarial learning is highly sensitive to hyperparameter settings and network balancing, often resulting in unstable convergence. In addition, the generation and refinement of volumetric or point cloud data impose significant computational and memory demands, which restrict scalability to larger or higher-resolution datasets. The Mixture of Experts (MoE) architecture has recently attracted considerable attention, particularly within the domains of large language models (LLMs) and neural networks. A limited number of studies have explored the integration of MoE with GANs to address the task of 3D object generation. For example, [ 24 ] introduced MEGAN, a novel generative architecture specifically designed to capture and model the complex modalities inherent in datasets. Furthermore, MoE has been shown to enhance the representational capacity of generators in sequence GANs, particularly when combined with the Feature Statistics Alignment (FSA) framework, which delivers informative learning signals that improve generator training [ 25 ] . Also, a novel hierarchical mixture of generators that extends the MoE concept to generative modeling is in presented in [ 26 ] . The model leverages a tree structure with soft decision splits and localized generators at the leaves, enabling hierarchical clustering within the generative process. Its continuity and compatibility with gradient-based optimization distinguish it as a flexible and scalable alternative to standard GANs. In addition, [ 27 ] proposed an innovative method that leverages a mixture of multiple noise distributions to address the challenge of aligning GAN-generated outputs with target data distributions that are either unknown or significantly different from the input noise. III Method This section outlines the core structure and design principles behind our proposed MoE-CGAN framework for 3D object generation and completion. We present the essential elements of our CGAN architecture, the MoE framework with its enhanced gating mechanism, expert networks, and the routing strategy that dynamically distributes inputs among specialized experts (see Figure 2 ). Subsequently, we describe our novel training strategy employing DCC for stable and efficient optimization. Fig. 2: The proposed architecture of MoE-CGAN. The gating network GN takes both the latent vector \mathbf{z} and partial input \mathbf{x_{p}} (for completion tasks) to route to specialized experts. III-A CGAN Architecture with Enhanced 3D Representations Our baseline CGAN architecture employs deep and complex layers for both Generator and Discriminator, specifically designed to capture intricate features in 3D data. To address the limitations of low-resolution voxel representations, we explore two enhanced architectures: Generator (G) : The generator uses a combination of convolutional and transposed convolutional layers with residual connections to mitigate gradient vanishing. It begins with 3D convolutional layers ( Conv3d ) using kernel size 4 and stride 2 for downsampling, each followed by batch normalization ( BatchNorm3d ) and ReLU activation. The architecture progressively increases filter channels (from 64 to 512) while decreasing spatial dimensions. Dilated convolutions with increasing rates (2, 4, 8) capture multi-scale features without additional spatial reduction. For upsampling, transposed convolutional ( ConvTranspose3d ) layers with skip connections reconstruct high-resolution features. For enhanced representation, we implement two variants: a sparse convolutional variant and a hybrid output representation. In the first variant, we employ the Minkowski Engine for sparse tensor operations, which enables high-resolution voxel processing (up to 128^{3} ) while preserving computational efficiency. In the second, hybrid representation, the final layer produces either conventional voxel grids or a triplane representation \mathbf{T}\in\mathbb{R}^{3\times D\times D} , which can be decoded into high-resolution implicit surfaces using a lightweight MLP decoder. Discriminator (D) : The discriminator employs multiple Conv3d layers with spectral normalization for training stability, reaching up to 512 filters. It downsamples inputs while applying instance normalization and LeakyReLU activations. The final convolutional output is flattened and passed through a linear layer with Sigmoid activation for real/fake classification. For the hybrid representation variant, the discriminator processes the triplane features through separate 2D convolutional pathways before fusion. III-B MoE Pipeline with Context-Aware Gating Our MoE architecture enhances GANs by dynamically routing input data to specialized expert networks, each trained to model distinct geometric patterns in 3D space. This modular approach improves computational efficiency through sparse activation while mitigating mode collapse by distributing complex data distributions across specialized generators. III-B 1 Expert Specialization Strategy We employ multiple expert generators \mathbf{G}={G_{1},G_{2},\ldots,G_{n}} , where each G_{i} shares the same base architecture but develops unique specialization through our training strategy. For a latent vector \mathbf{z}\sim\mathcal{N}(0,1) , each generator produces output voxels h_{i}\in\mathbf{H}={h_{1},h_{2},\ldots,h_{n}} and intermediate feature representations f_{i}\in\mathbf{F}={f_{1},f_{2},\ldots,f_{n}} extracted from the second transposed convolutional layer. III-B 2 Context-Aware Gating Networks The gating network \mathbf{GN} serves as the router that dynamically directs inputs to the most appropriate experts based on both latent characteristics and, for completion tasks, the partial input structure. This represents a key enhancement over standard MoE approaches by incorporating task-specific context. \mathbf{g}=\text{GN}(\mathbf{z},\mathbf{x_{p}},\mathbf{F}) (1) where \mathbf{x_{p}} represents the partial input voxel grid in completion case, enabling context-aware routing based on the missing regions that need completion. The gating mechanism computes affinity scores through a multi-layer perceptron with two hidden layers (512 and 256 units) with GeLU activations: s_{i}=\text{MLP}([\mathbf{z},\mathbf{x_{p}},f_{i}]),\quad\forall i\in{1,2,\ldots,n} (2) Scores are normalized using sparse softmax with learnable temperature scaling: p_{i}=\frac{\exp(s_{i}/\tau)}{\sum_{j=1}^{n}\exp(s_{j}/\tau)} (3) We employ Top-K selection with adaptive K based on input complexity: g_{i}=\begin{cases}p_{i}&\text{if }i\in\text{top}_{k}(\mathbf{p})\\ 0&\text{otherwise}\end{cases} (4) The final output is computed as: \text{GI}=\sum_{i=1}^{n}g_{i}h_{i} (5) This enhanced gating enables specialized completion where different experts focus on distinct geometric structures (e.g., planar surfaces, curved elements, fine details) based on the missing regions in \mathbf{x_{p}} . III-B 3 Training Strategy with Dynamic Specialization Our training strategy employs an auxiliary-loss-free approach with DCC (see Algorithm 1) that promotes implicit expert specialization while maintaining training stability. Algorithm 1 MoE Training with Context-Aware DCC 1: Generator G ; discriminator D ; gating network GN ; Training dataset \mathcal{X} ; latent dimension d ; base capacity C_{\text{b}} ; momentum \mu ; adjustment rate \alpha ; batch size B . 2: U_{\text{avg}}\leftarrow\mathbf{0} , C_{i}\leftarrow C_{\text{b}} 3: for each training iteration do 4: Sample latent vectors z\sim\mathcal{N}(0,1)^{d} , real data x\sim\mathcal{X} , partial inputs x_{p}\sim\mathcal{X}{\text{partial}} 5: Initialize expert load counter: L\leftarrow\mathbf{0} 6: Compute context-aware gating weights: 7: for each sample (z_{b},x{p_{b}}) in batch do 8: \text{indices}=\text{top-k}(GN(z_{b},x_{p_{b}},\mathbf{F})) 9: for each candidate i in indices do 10: if L[i]