MoE3D: A Mixture-of-Experts Module for 3D Reconstruction

Zichen Wang Ang Cao Liam J. Wang Jeong Joon Park [email protected] [email protected] [email protected] [email protected] University of Michigan, Ann Arbor · 2026

Read MoE3D: A Mixture-of-Experts Module for 3D Reconstruction by Zichen Wang Ang Cao Liam J. Wang Jeong Joon Park [email protected] [email protected] [email protected] [email protected] University of Michigan, Ann Arbor (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05208.txt.

Introduction

Zichen Wang and 3 co-authors add a mixture-of-experts module to feed-forward 3D reconstruction in a 2026 paper. Models that rebuild a scene's depth from images struggle near depth discontinuities, the sharp edges where one surface ends and another begins. Standard regression losses encourage averaging there, blurring exactly the boundaries that define objects, rooms, and obstacles.

Their module combines multiple smooth depth predictions, with a softmax weighting head choosing among the hypotheses pixel by pixel. Integrated into a pretrained state-of-the-art 3D model, it achieves substantial improvement near boundaries. Sharper depth edges matter for robotics, augmented reality, and scanning, where a blurred boundary can mean a misjudged doorway, shelf edge, or grasp point.

Catalog record

CREATORZichen Wang Ang Cao Liam J. Wang Jeong Joon Park [email protected] [email protected] [email protected] [email protected] University of Michigan, Ann Arbor
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-05208

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoE3D: A Mixture-of-Experts Module for 3D Reconstruction Zichen Wang Ang Cao Liam J. Wang Jeong Joon Park [email protected] [email protected] [email protected] [email protected] University of Michigan, Ann Arbor Abstract Figure 1 : MoE3D is a mixture-of-experts module designed to sharpen depth boundaries and mitigate flying-point artifacts (highlighted in red) of existing feed-forward 3D reconstruction models (left side). MoE3D predicts multiple candidate depth maps and fuses them via dynamic weighting (visualized by MoE weights on the right side). When integrated with a pre-trained 3D reconstruction backbone such as VGGT, it substantially enhances reconstruction quality with minimal additional computational overhead. Best viewed digitally. Abstract We propose a simple yet effective approach to enhance the performance of feed-forward 3D reconstruction models. Existing methods often struggle near depth discontinuities, where standard regression losses encourage spatial averaging and thus blur sharp boundaries. To address this issue, we introduce a mixture-of-experts formulation that handles uncertainty at depth boundaries by combining multiple smooth depth predictions. A softmax weighting head dynamically selects among these hypotheses on a per-pixel basis. By integrating our mixture model into a pre-trained state-of-the-art 3D model, we achieve substantial reduction of boundary artifacts and gains in overall reconstruction accuracy. Notably, our approach is highly compute efficient, delivering generalizable improvements even when fine-tuned on a small subset of training data while incurring only negligible additional inference computation, suggesting a promising direction for lightweight and accurate 3D reconstruction. 1 Introduction Feed-forward 3D reconstruction models, such as DUSt3R 35 and VGGT 32 , have shown impressive flexibility, accuracy, and efficiency. These models are typically trained in a regression-based manner to predict depth or point maps. However, depth boundaries often exhibit abrupt discontinuities that introduce substantial uncertainty in depth estimation. When simple regression losses are used, these models tend to blur these boundaries to minimize large penalties from sharp prediction errors, resulting in common flying-point artifacts and overly smooth predictions (Fig. 1 ). Although generative training schemes such as GANs or diffusion models can better capture uncertainty, these approaches entail significant computational overhead during both training and inference. In this work, we introduce a lightweight module, MoE3D, that effectively models prediction uncertainty with minimal additional computational cost when attached and fine-tuned on a pre-trained VGGT. MoE3D adopts a mixture-of-experts design, producing multiple depth predictions and corresponding weights from several output heads. These predictions are fused through a mixture model formulation with a softmax weighting on per-pixel basis. By generating multiple hypotheses, the model can better handle multi-modal distributed depths near boundaries. We integrate our MoE3D module into the recent 3D reconstruction network VGGT, which achieves state-of-the-art performance in depth, point, and camera pose prediction. Specifically, we attach the MoE module to VGGT’s depth prediction head and fine-tune it on a small subset of the original training data to assess its impact. By combining multiple expert heads with entropy-based regularization, our model naturally develops strong specialization near depth boundaries (see Fig. 3 ). As a result, MoE3D substantially sharpens boundary regions, enhances overall reconstruction quality, and pushes the performance frontier of current feed-forward 3D reconstruction methods, while introducing only a modest computational overhead of approximately 7% during inference. In monocular depth estimation, our module maintains the prediction accuracy of VGGT while markedly improving boundary sharpness and precision (Tab. 2 ). Moreover, on multi-view 3D reconstruction, MoE3D boosts 3D prediction accuracy by more than 30% on indoor scenes (Tab. 1 ), making it the leading feed-forward reconstruction system. Overall, our mixture-of-experts framework provides a simple yet effective solution to the pervasive problem of boundary uncertainty in modern 3D reconstruction models, substantially improving both their accuracy and perceptual quality. While our experiments focus on VGGT, the current state-of-the-art 3D model, the same principle can readily extend to other feed-forward architectures that suffer from uncertainty-induced blurring, pointing toward a promising new direction for efficient and accurate 3D reconstruction. 2 Related Works Feed-Forward 3D Reconstruction. Early 3D reconstruction methods, such as Structure-from-Motion (SfM) and Multi-View Stereo (MVS) 26 ; 41 , rely on geometric optimization over correspondences and camera parameters. Recent transformer-based approaches reformulate this process as direct feed-forward regression of geometric attributes. DUSt3R 35 and MASt3R 18 first demonstrated that a pair of unposed images can be mapped to dense, aligned pointmaps, removing the need for explicit triangulation. VGGT 32 further generalized this idea to handle dozens of views with a single large transformer, jointly predicting cameras, depth maps, and point maps in a single forward pass. Subsequently, several variants 38 ; 36 ; 16 ; 31 ; 44 explored various architectural modifications for scalability, dynamic scenes, or online inference. These works establish the foundation of feed-forward 3D reasoning, but their predictions remain spatially smooth, often oversmoothing depth discontinuities and object boundaries due to the continuous nature of regression losses. Depth Estimation and Boundary Sharpness. Single-view depth estimation has been explored through both discriminative and generative approaches 23 ; 39 ; 15 . Discriminative methods 23 ; 39 ; 40 ; 33 ; 34 achieve strong zero-shot generalization through large-scale multi-dataset pretraining, while generative approaches 15 ; 13 finetune pretrained diffusion models to leverage rich visual priors to synthesize depth maps. Despite their success, both approaches struggle with flying points near object boundaries. Discriminative models tend to average across depth discontinuities under \ell_{1}/\ell_{2} regression losses, while generative methods typically rely on low-dimensional latent representations (e.g., VAE bottlenecks) that compromise structural detail. Recent works such as Depth Pro 1 and Pixel-Perfect Depth 37 explicitly target this issue with boundary-aware losses and pixel space diffusion. Our method instead addresses boundary sharpness from an architectural perspective: by introducing a Mixture-of-Experts (MoE) head that enables spatial specialization among experts, we preserve feed-forward efficiency while achieving sharper and more geometrically consistent predictions. Layered and Multi-Hypothesis Depth. Closely related to mixture-based depth modeling is a long line of work on layered and multi-hypothesis depth representations. Early graphics work such as Layered Depth Images (LDI) 28 represents scenes using multiple depth layers per pixel to explicitly capture occlusions and visibility, and has since inspired learning-based extensions. Subsequent methods extend to layered depth prediction from a single image 9 , layered stereo representations 45 , and multi-hypothesis optimization in classical multi-view stereo 5 . In parallel, mixture-density–based approaches parameterize depth ambiguity probabilistically, predicting multiple depth modes with learned mixing weights 30 ; 42 ; 43 ; 6 . While these methods explicitly maintain multiple depth layers or mixture components to handle occlusions and depth ambiguity, hypothesis combination is typically performed via parametric distribution heads, hand-designed selection rules, or downstream probabilistic inference. In contrast, our method performs per-pixel, learned routing between multiple feed-forward depth experts within a unified network, enabling end-to-end specialization and near-hard selection without requiring explicit layered representations or parametric mixture modeling. To our knowledge, this work is the first to introduce a mixture-of-experts formulation into feed-forward multi-view depth networks, bridging classical multi-hypothesis depth reasoning with modern MoE architectures. Mixture-of-Experts for Vision and Geometry. Mixture-of-Experts (MoE) architectures 17 ; 11 ; 10 ; 22 were originally developed for language models to scale model capacity 11 ; 10 ; 22 . Vision variants route tokens or regions to experts for efficiency or diversity 8 ; 24 ; 7 . Our design draws inspiration from these models but departs in both scope and

Read the complete text