LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts
Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, Bita Rouhani · 2026
Read LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts by Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, Bita Rouhani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-18089.txt.
Introduction
Venmugil Elango and co-authors ask how close today's mixture-of-experts models are to the best possible use of compute. Their 2026 preprint studies MoE designs, now central to top open and closed language models. An MoE routes each token to a few expert parts, so cost per token stays low. Yet no one knew how much accuracy each unit of compute or each parameter could in principle buy. The authors frame that gap as accuracy per FLOP and per parameter.
Their study, named LatentMoE in the title, probes where the waste sits. Knowing the ceiling helps designers spend on the right parts. At the scale of modern training runs, even small gains save vast sums.
Catalog record
| CREATOR | Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, Bita Rouhani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-18089
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts Venmugil Elango, Nidhi Bhatia, Roger Waleffe, Rasoul Shafipour, Tomer Asida, Abhinav Khattar, Nave Assaf, Maximilian Golub, Joey Guman, Tiyasa Mitra, Ritchie Zhao, Ritika Borkar, Ran Zilberstein, Mostofa Patwary, Mohammad Shoeybi, Bita Rouhani Abstract Abstract. Mixture of Experts (MoEs) have become a central component of many state-of-the-art open-source and proprietary large language models. Despite their widespread adoption, it remains unclear how close existing MoE architectures are to optimal with respect to inference cost, as measured by accuracy per floating-point operation and per parameter. In this work, we revisit MoE design from a hardware-software co-design perspective, grounded in empirical and theoretical considerations. We characterize key performance bottlenecks across diverse deployment regimes, spanning offline high-throughput execution and online, latency-critical inference. Guided by these insights, we introduce LatentMoE , a new model architecture resulting from systematic design exploration and optimized for maximal accuracy per unit of compute. Empirical design space exploration at scales of up to 95B parameters and over a 1T-token training horizon, together with supporting theoretical analysis, shows that LatentMoE consistently outperforms standard MoE architectures in terms of accuracy per FLOP and per parameter. Given its strong performance, the LatentMoE architecture has been adopted by the flagship Nemotron-3 Super and Ultra models and scaled to substantially larger regimes, including longer token horizons and larger model sizes, as reported in 13 . 1 Introduction Transformer-based large language models underpin a wide range of modern AI systems, from conversational agents to code generation and scientific reasoning. As these models continue to scale, practical deployment is increasingly constrained by inference cost, encompassing both computation and memory. As a result, a central objective in modern model design is to maximize achievable accuracy under fixed inference cost constraints. Mixture-of-Experts (MoE) architectures have emerged as a promising approach towards this goal, enabling models to scale parameter count while keeping the number of Floating-point Operations (FLOPs) per token fixed. Despite their empirical success, the MoE design space remains poorly understood. Existing MoE architectures are largely motivated by high-level sparsity arguments and are optimized primarily for offline, throughput-oriented settings, with limited consideration of online deployments that impose strict latency, memory bandwidth, and communication constraints. We argue that effective MoE design must be evaluated along two complementary dimensions: accuracy per FLOP and accuracy per parameter. While accuracy per FLOP captures computational efficiency, accuracy per parameter reflects memory footprint, memory bandwidth demands, routing-induced communication, and sharding overheads (factors that are often the dominant bottlenecks in interactive, low-latency inference). Neglecting these factors can lead to architectures that appear efficient in aggregate compute, yet incur substantial inefficiencies in practical deployment. (a) Standard MoE architecture. (b) LatentMoE architecture. Figure 1 : Standard MoE vs. LatentMoE architectures. In LatentMoE, tokens are projected from the model hidden dimension d into a smaller latent dimension \ell for expert routing and computation, which reduces routed parameter loads and all - to - all traffic by a factor of d/\ell . We use this efficiency to increase the total number of experts and the top-k active experts per token by the same factor d/\ell , which improves the accuracy of the model while keeping overall inference cost approximately constant. In this work, we revisit MoE architecture design from a hardware–software co-design perspective. Through a systematic analysis of existing MoE systems across the throughput–latency Pareto frontier, we identify key structural bottlenecks arising from expert parameterization, routing-induced all-to-all communication, and memory access patterns. Combined with detailed accuracy measurements and theoretical analysis, our study identifies structural inefficiencies in prevailing MoE designs that limit achievable accuracy per unit of inference cost. Guided by these insights, we introduce LatentMoE, a new mixture-of-experts architecture explicitly optimized for both accuracy per FLOP and accuracy per parameter. LatentMoE decouples expert routing and computation from the model hidden dimension by projecting incoming activations into a shared low-dimensional latent space prior to expert processing (Figure 1 ). The latent dimension serves as a direct control knob for computational cost, communication volume, and expert parameter size. At iso-FLOP and iso-parameter count, projecting incoming activations into a lower-dimensional latent space enables a proportional increase in both the number of experts and the routing top-k, while maintaining constant inference cost. As we show both theoretically and empirically, simultaneously increasing expert count and combinatorial sparsity diversity improves the effective expressivity of the model, leading to higher achievable accuracy. Crucially, these gains arise without increasing memory bandwidth demands or communication overheads, making LatentMoE well suited for both latency-critical and throughput-oriented deployments. We validate the LatentMoE concept through pretraining experiments at scales of up to 95B parameters and over 1T tokens. Across all evaluated regimes, LatentMoE consistently improves upon standard MoE architectures, achieving higher accuracy at fixed inference cost or substantially lower inference cost at fixed accuracy. Given its strong performance, the LatentMoE architecture has been adopted by the flagship Nemotron-3 Super and Ultra models and scaled to substantially larger regimes, including longer token horizons and larger model sizes ( 13 ) . 2 LatentMoE Core Design Principles Before delving into the specifics of LatentMoE, we first take a systems-level view of what is required to deploy an MoE model that is both accurate and cost-efficient. Throughout this section, we use Qwen3-235B-A22B as a running example for our modeling, with N=128 experts, K=8 active experts per token, a hidden dimension d=4096 , and an intermediate feed-forward dimension m=1536 . For concreteness, we consider deployment on NVIDIA GB200 GPUs interconnected by a high-bandwidth NVLink fabric, which provides approximately 1800 GB/s of bidirectional bandwidth per GPU (i.e., \mathrm{BW}_{\mathrm{NVL}}=900 GB/s per direction). To ensure that expert communication remains within a single NVLink domain, experts are distributed via expert parallelism across \mathrm{EP}=64 GPUs. Attention layers are executed using data parallelism over the same group of GPUs. Each GB200 GPU delivers a peak FP4 Tensor Core throughput of F=10 PFLOPs and an HBM memory bandwidth of \mathrm{BW}_{\mathrm{HBM}}=8 TB/s ( 14 ) . 2.1 Memory Bandwidth Bottleneck In highly interactive ( i.e . , low-latency) settings that typically use small batch sizes, MoE computation is primarily bottlenecked by memory bandwidth. Figure 2 provides a high-level roof-line analysis of performance versus arithmetic intensity. Figure 2 : Roofline Analysis for serving Qwen3-235B-A22B. Operating points correspond to different per-expert token counts t_{\mathrm{exp}} (i.e., effective expert batch sizes after MoE routing), mapped to arithmetic intensity I=\frac{2\cdot t_{\mathrm{exp}}\cdot d\cdot m}{d\cdot m+t_{\mathrm{exp}}\cdot(d+m)} . At latency-critical batch sizes (low I ), MoE expert computation is constrained by HBM bandwidth rather than compute, and the operating points lie in the bandwidth-bound regime. For a GB200 system, a computation becomes compute-bound only if its arithmetic intensity (i.e., FLOPs per byte) exceeds: \frac{F}{\mathrm{BW}_{\mathrm{HBM}}}=\frac{10\times 10^{15}}{8\times 10^{12}}=1250\text{ FLOPs/byte}. Let t_{\mathrm{total}} be the total number of tokens across the \mathrm{EP} GPUs prior to MoE routing. Assuming a uniform distribution of tokens across experts, the number of tokens assigned to a single expert is: t_{\mathrm{exp}}:=\frac{t_{\mathrm{total}}\cdot K}{N} . In the Qwen3-235B-A22B example, with N=128 and \mathrm{EP}=64 , each GPU hosts N/\mathrm{EP}=2 experts; thus each GPU processes approximately 2\cdot t_{\mathrm{exp}} expert tokens per MoE layer. The FP4 compute cost for a single expert is C_{\mathrm{exp}}=2\cdot t_{\mathrm{exp}}\cdot d\cdot m , and the corresponding memory traffic in FP4 precision—accounting for weights, inputs, and intermediate activations—is given by M_{\mathrm{exp}}=d\cdot