Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers: Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock
Haiying Sha · 2026
Read Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers: Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock by Haiying Sha (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19378.txt.
Introduction
Sha diagnoses why token-choice sparse expert models fail when a five-billion-parameter dense video diffusion model is converted to experts. Linear routers saturate and homogenize experts, MLP routers deadlock a third of layers on one expert, and cross-attention routers recover only partly. Deadlocked layers cluster at shallow visual and deep semantic depths, and half-precision arithmetic wipes out tiny updates. Sha reads deadlock as a waiting strategy before shared experts mature, and sets out conversion laws plus a fix for the precision trap.
Catalog record
| CREATOR | Haiying Sha |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74134 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers: Diagnosis, Boundary Calibration and Evolutionary Roadmap from Routing Collapse to Selective Deadlock Haiying Sha Abstract Mixture-of-Experts (MoE) has become a core paradigm for scaling model capacity in language modeling, yet its application to video generation with Diffusion Transformers (DiTs) remains nascent, and existing attempts have yielded limited benefits. This paper systematically diagnoses failure modes in the training dynamics of Token-Choice sparse MoE for video diffusion models. Starting from a pretrained dense video generation model (about 5B parameters), we convert it into an MoE architecture: routed experts are exactly cloned from the original dense FFN weights, shared experts are initialized with extremely small non-zero noise (e.g., scale 10 − 4 10^{-4} ; a zero-initialized variant is used solely for verification to guarantee output equivalence), and only the routers start learning from random initialization. Experiments reveal a hierarchy of critical failures: (1) linear routers suffer global soft saturation, with vanishing Softmax gradients leading to complete expert homogenization (cosine similarity > > 0.99); (2) non-linear MLP routers resolve global deadlock but introduce a more subtle phenomenon, selective deadlock , where roughly one-third of layers degenerate into a single-expert mode and increasing the auxiliary loss to 0.2 cannot prevent it; (3) routers that fuse multimodal features via cross-attention exhibit preliminary self-recovery—the only observed instance of a deadlocked layer spontaneously rebounding—yet about 9 layers remain stubbornly deadlocked; (4) the distribution of these deadlocked layers follows a pronounced U-shaped structure : heavily concentrated in shallow visual processing layers (1–10) and deep semantic integration layers (18–25), while intermediate layers maintain near-uniform utilization; (5) bfloat16 mixed precision causes tiny expert weight updates to be truncated to zero by hardware—an easily overlooked numerical trap. By tracking the routing decision time series of over 65 million tokens across 30 Transformer layers over 5,000 training steps, we establish a complete phenomenology of deadlock. We interpret this phenomenon as functional redundancy dynamics within the trilateral system of “gate, shared expert, routed experts” and propose the Functional Redundancy Hypothesis —which predicts that a zero/minimally initialized and insufficiently trained shared expert must reach a critical capability threshold before the system can release the silenced routed experts. This hypothesis aligns with the theory of functional redundancy in systems biology, offering a novel theoretical perspective for understanding MoE training dynamics. On the engineering side, we document the Three Laws of Dense-to-MoE conversion (expert structural consistency, 1:1 weight cloning, and shared expert verification-zero/training-micro-noise initialization), together with a complete solution for the bfloat16 precision trap. Building on this, we further explore the In-Context Conditioning paradigm for multimodal features, an MoE-based text rendering solution (introducing a dedicated text expert + Glyph-ByT5 glyph encoder), and an evolutionary path toward joint audio-video generation. Based on these diagnostic findings, we lay out a three-step evolutionary roadmap from “visual unification” to “general intelligence engine”: near-term targeted enhancement through MoE-ification, mid-term cross-modal audio-video generation by adding audio experts, and long-term integration of physical priors and causal reasoning to build a world model that empowers embodied intelligence. Core thesis : Under the current Token-Choice paradigm, deadlock in a subset of layers is structural and independent of router complexity; completely eliminating dead experts in visual MoEs requires starting from the initialization strategy of shared experts, differentiated design of expert weights, or an upgrade of the routing paradigm itself. This work represents one of the first comprehensive empirical diagnoses of routing failure in real-world video generation MoE deployments, providing clear guidance for future research. \triangleright GitHub https://github.com/Shybert-AI/UniGen-MOE 1 Introduction 1.1 Background and Motivation Mixture-of-Experts (MoE) [ 1 , 2 ] has become a core paradigm for scaling the capacity of deep learning models. Its basic idea—dynamically routing each input token to a set of specialized “expert” sub-networks via a sparse gating mechanism—has achieved extraordinary success in language modeling. Systems such as DeepSeek-V4 (1.6 trillion parameters, 384 experts per layer, 6 activated per token) [ 3 ] , the Qwen3 series [ 4 ] , and Kimi K2.6 all adopt MoE architectures, realizing orders-of-magnitude leaps in parameter count and performance while maintaining computational efficiency. The success of these systems rests on a key assumption of expert specialization: different sub-networks learn to process non-overlapping, semantically meaningful regions of the input space. However, the exploration of applying MoE to visual generation—especially video Diffusion Transformers (DiTs)—is still in its early stages. Although works such as ProMoE [ 5 ] , DiffMoE [ 6 ] and Mamoda2.5 [ 7 ] have introduced MoE modules into diffusion Transformers, a comprehensive failure diagnosis is lacking. We argue that there are fundamental differences between visual MoEs and language MoEs: language tokens are highly semantically condensed and discrete, whereas visual patch tokens exhibit extreme spatial redundancy [ 5 , 8 ] , distribution drift induced by diffusion timesteps [ 9 ] , and conflicting routing signal sources in multi-task settings [ 10 ] . These differences subject the routing mechanisms matured in the language domain to systematic and uncharted risks in visual generation. 1.2 Research Questions and Main Contributions This paper aims to systematically diagnose and calibrate the failure modes of Token-Choice sparse MoEs deployed on video Diffusion Transformers. We start from the pretrained Kiwi-Edit video editing model, which consists of the Wan2.2-TI2V-5B video diffusion backbone and the Qwen2.5-VL-3B-Instruct multimodal encoder (totaling about 8B parameters). We convert it into an MoE using three laws: preserving expert structural consistency, 1:1 weight cloning for routed experts, and an initialization strategy where shared experts are set to zero during the verification phase and enabled with micro-noise (e.g., 10^{-4} ) for training. We then let the routers learn expert assignments from scratch. The key features of the experimental setup—extremely low-noise shared experts, two weight-cloned routed experts, and a 9-in-1 [ 17 ] multi-task video generation scenario—create an ultra-rigorous trial where no routing failure can be masked by pretrained capabilities. Our main contributions are as follows: 1. First comprehensive diagnosis : We provide a systematic classification of failure modes of Token-Choice sparse MoE in video diffusion models, spanning a full architectural evolution comparison from linear routers to cross-attention routers, and establish a complete phenomenology of deadlock. 2. Discovery of “selective deadlock” : We identify and name a distinct failure mode that emerges after global deadlock is eliminated, driven by trilateral interactions among the gate, shared experts, and routed experts. It manifests as roughly 1/3 of layers degenerating into a single-expert usage pattern that cannot be prevented by increasing the auxiliary loss. 3. Discovery of the “U-shaped deadlock distribution” : Using temporal data from 30-layer networks with over 65 million tokens, we precisely map for the first time the structural distribution of deadlocked layers—heavily concentrated in shallow visual processing layers and deep semantic integration layers, with intermediate layers remaining balanced. 4. Proposal of the “Functional Redundancy Hypothesis” : Based on routing decision time-series data over 5,000 steps and more than 3.4 billion tokens, we reinterpret deadlock as a rational, phased resource allocation strategy. This hypothesis finds theoretical grounding in systems biology and offers a novel theoretical perspective for understanding MoE training dynamics. 5. Engineering contributions : We document the Three Laws of Dense-to-MoE conversion, the precision trap of MoE expert weight updates under bfloat16 mixed-precision training and its complete solution, and clarify the initialization paradigm