Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelism Chenwei Cui, Rockwell Jackson, Benjamin Joseph Herrera, Ana María Tárano, Hannah Kerner Abstract Large language models have transformed many applications but remain expensive to train. Sparse Mixture of Experts (MoE) addresses this through conditional computation, with Expert Parallel (EP) as the standard distributed training method. However, EP has three limitations: communication cost grows linearly with the number of activated experts k k , load imbalance affects latency and memory usage, and data-dependent communication requires metadata exchange. We propose Multi-Head LatentMoE and Head Parallel (HP), a new architecture and parallelism achieving O ⁡ ( 1 ) O(1) communication cost regardless of k k , completely balanced traffic, and deterministic communication, all while remaining compatible with EP. To accelerate Multi-Head LatentMoE, we propose IO-aware routing and expert computation. Compared to MoE with EP, Multi-Head LatentMoE with HP trains up to 1.61 × 1.61\times faster while having identical performance. With doubled granularity, it achieves higher overall performance while still being 1.11 × 1.11\times faster. Our method makes multi-billion-parameter foundation model research more accessible. https://github.com/kerner-lab/Sparse-GPT-Pretraining † † affiliationnotice: Equal contribution 1 Introduction Large language models have transformed applications such as code generation ( 2 ) , but training them remains expensive ( 1 ) . Pretraining a large model consumes substantial compute and electricity ( 3 ) . These costs limit who can conduct foundation model research. Reducing training costs while maintaining model quality has become a key interest in machine learning research. The sparse Mixture of Experts (MoE) architecture addresses this challenge through conditional computation ( 14 ) . For each input token, a router activates only a small subset of expert networks. This enables models to scale capacity without proportionally increasing compute ( 25 ; 10 ) . Expert Parallel (EP) is the standard distributed training method for MoEs ( 18 ; 10 ) . Each token is routed to k experts, replicated k times, and sent to corresponding GPUs via all-to-all communication. Results are processed by experts and returned via another all-to-all operation ( 26 ) . EP has three limitations. First, both communication volume and all-to-all latency are O(k) . Second, expert load imbalance further degrades all-to-all performance. Third, non-deterministic communication necessitates an additional all-to-all to exchange metadata ( 26 ) . Recently, LatentMoE ( 9 ) addressed the communication bottleneck by projecting tokens from hidden dimension d into a smaller latent dimension d_{h} before expert routing and computation. This reduced both per-expert parameter loads and all-to-all traffic by a factor of \frac{d}{d_{h}} , enabling higher k values at similar communication costs. However, LatentMoE still relied on Expert Parallel; communication happened after routing, so load imbalance and non-deterministic patterns persisted. We propose Multi-Head LatentMoE and Head Parallel (HP) to address these limitations. Multi-Head LatentMoE decomposes a single MoE into multiple independent smaller modules, splitting each input token into N_{h} sub-tokens. Each sub-token is processed by an independent MoE module with its own router and experts. HP exploits this structure. The sub-tokens are distributed to GPUs before any routing decision. Each GPU completes all routing and expert computation locally. This has three advantages over EP: 1. Communication volume is constant because each token is sent exactly once before routing. 2. Traffic is balanced because sub-tokens are evenly distributed regardless of routing decisions. 3. The inter-GPU communication is deterministic because it does not depend on routing decisions. A naive implementation of Multi-Head LatentMoE would multiply High Bandwidth Memory (HBM) usage and IO cost by N_{h} , because it materializes the full routing scores and expert activations for each head. We develop IO-aware routing by doing online top- k directly in SRAM, reducing HBM access from O(T\cdot N_{e}+N_{e}) to O(T+N_{e}) , where T is the number of tokens and N_{e} is the number of experts. We also develop IO-aware expert computation by formulating expert computation as block-sparse attention and leveraging FlexAttention ( 8 ) , reducing HBM access from O(T\cdot d_{e}+d_{e}) to O(T+d_{e}) , where d_{e} is the number of neurons per expert. Both algorithms are exact. We evaluate Multi-Head LatentMoE with HP on language modeling using 10B tokens from FineWebEdu ( 22 ) . Our method trains up to 1.61x faster than the standard MoE with EP while having identical performance. The inter-GPU communication volume is reduced to 25% when k=4 . Under the same number of active parameters, our method can reach 6.9 percentage points higher overall accuracy than MLP. When doubling the granularity, our method achieves higher overall model performance while still being 1.11\times faster. Our contributions are: 1. We propose Multi-Head LatentMoE and Head Parallel, achieving O(1) communication volume for any k , perfect load balance, and deterministic communication patterns for distributed MoE training. 2. We develop exact IO-aware routing and expert computation operations, making Multi-Head LatentMoE practical and efficient. 3. We demonstrate that Multi-Head LatentMoE with HP trains 1.61x faster than standard MoE with EP. With two times the granularity, it further achieves higher overall accuracy while still being 1.11\times faster. 4. Our work accelerates ultra-sparse MoE pre-training, making multi-billion-parameter foundation model research more accessible to the academic research community. 2 Prerequisites 2.1 Notation We use italic letters for scalars, bold lowercase for vectors, and bold uppercase for matrices or tensors. B denotes batch size, T denotes the number of tokens in a sequence, and t indexes token position. The model hidden dimension is d , while N_{h} and d_{h} denote the number of heads and per-head dimension, respectively. For mixture-of-experts layers, N_{e} is the number of experts, k is the number of activated experts per token, and d_{e} is the expert hidden dimension. The input and output tokens at position t are \mathbf{x}_{t}\in\mathbb{R}^{d} and \mathbf{o}_{t}\in\mathbb{R}^{d} , with projection matrices \mathbf{W}_{\text{in}},\mathbf{W}_{\text{out}}\in\mathbb{R}^{d\times d} . 2.2 Mixture of Experts Mixture of Experts (MoE) scales model capacity without proportionally increasing computational cost ( 25 ) . This definition follows Mixtral ( 17 ) . We re-use this for our baseline, as well as to help define Multi-Head LatentMoE. An MoE has N_{e} neural network modules called experts. Among them, k experts are activated for each token ( 25 ) . For each input token \mathbf{x}_{t} , the corresponding output \mathbf{o}_{t} is: \displaystyle\mathbf{o}_{t} \displaystyle=\sum_{i=1}^{N_{e}}g_{i,t}E_{i}(\mathbf{x}_{t}), (1) \displaystyle g_{i,t} \displaystyle=\frac{\exp(g^{\prime}_{i,t})}{\sum_{j=1}^{N_{e}}\exp(g^{\prime}_{j,t})}, (2) \displaystyle g^{\prime}_{i,t} \displaystyle=\begin{cases}s_{i,t},&s_{i,t}\in\text{top-}k(\{s_{j,t}|1\leq j\leq N_{e}\})\\ -\infty,&\text{otherwise}\end{cases}, (3) \displaystyle s_{i,t} \displaystyle=r(\mathbf{x}_{t})_{i}. (4) where g_{i,t} is the normalized gating value for the i -th expert; E_{i}(\cdot) denotes the i -th expert; s_{i,t} and g^{\prime}_{i,t} are intermediate values; \text{top-}k(\cdot) selects the top k values from the set; and r(\cdot) is a linear mapping called the router. To ensure router stability, our load balancing strategy is global ( 23 ) and auxiliary-loss-free (aux-free) ( 29 ) . Following ( 19 ) , we replace the first two MoE layers with Multilayer Perceptrons (MLPs) to mitigate router imbalance in early layers. The router computation is performed in FP32. Figure 1 : Comparison of feedforward architectures. (a) Standard MLP applies a single feedforward network to each token. (b) Standard MoE uses a router to dynamically select experts from a single set. (c) LatentMoE performs routing first, then applies linear down-projection before expert computation and linear up-projection afterward. (d) Multi-Head LatentMoE projects each token into multiple sub-tokens, each processed by an independent MoE module with its own separately-trained router and expert set. Orange blocks denote activated experts; gray blocks denote inactive experts. Black lines indicate data flow. 2.3 Expert Parallel Expert Parallel (EP) distributes expert weights across GPUs, enabling training of large MoE models whose parameters exceed what a single GPU can hold ( 18 ) . The typical execution first duplicates tokens and clusters them by target experts ( 26 ) . Metadata about queue lengths are exchanged via all-to-all. Afterwards, tokens are dispatched to destination GPUs through token all-to-all. Received tokens are sorted by expert index, 1 1 1 For example, using moe_sort_chunks_by_index(.) from NVIDIA Transformer Engine ( 20 ) . for Grouped GEMM computation. Finally, outputs are re-sorted and returned via another all-to-all, then sorted again for weighted aggregation. EP has three limitations. First, communication volume increases with k due to token duplication. Second, if expert queues have uneven lengths, all-to-all must wait for the longest queue. Third, non-deterministic communication patterns require exchanging metadata via all-to-all. 2.4 Hardware-Aware Design Modern GPUs feature hierarchical memory with high-bandwidth low-capacity on-chip SRAM and low-bandwidth large-capacity off-chip HBM. The standard practice in architecture research is to exploit this structure. FlashAttention ( 7 ) uses tile streaming and recomputation to keep intermediate results in SRAM without materializing the full attention matrix in HBM. FlexAttention ( 8 ) extends FlashAttention with block sparse attention and addresses the software lottery problem, where developers must write and maintain custom CUDA programs for novel designs. We apply the same principle to implement IO-aware routing (see Section 3.3 ), keeping the intermediates in SRAM. For dropless ( 11 ) expert computation (see Section 3.4 ), we reuse FlexAttention to achieve highly optimized exact computation. 3 Method 3.1 Multi-Head LatentMoE The three limitations of Expert Parallel share a common root cause: the all-to-all traffic depends on routing decisions. We need an architecture that decouples all-to-all from routing. We propose Multi-Head LatentMoE (Figure 1 ). The key idea is to project a token into multiple sub-tokens, each processed by an independent MoE instance. This builds upon LatentMoE ( 9 ) and Multi-Head MoE ( 13 ) , but different from either. Specifically, for each input token \mathbf{x}_{t} , Multi-Head LatentMoE first projects and splits it into a list of sub-tokens: [\mathbf{x}_{t,1},\ldots,\mathbf{x}_{t,N_{h}}]=\text{split}(\mathbf{W}_{\text{in}}\mathbf{x}_{t}) (5) where \mathbf{W}_{\text{in}}\in\mathbb{R}^{d\times d} is a learned linear projection into the latent space, and \text{split}(\cdot) divides the projected vector into N_{h} sub-vectors of dimension d_{h} . The common practice ( 28 ) is to set d_{h}\cdot N_{h}=d , though they do not have to match. The output of Multi-Head LatentMoE is: \mathbf{o}_{t}=\mathbf{W}_{\text{out}}\cdot\text{concat}(f_{1}(\mathbf{x}_{t,1}),\ldots,f_{N_{h}}(\mathbf{x}_{t,N_{h}})) (6) where \mathbf{W}_{\text{out}}\in\mathbb{R}^{d\times d} projects the concatenated outputs back to the original space; each f_{i} is an independent MoE instance as defined in Section 2.2 , with its own router and expert set, sharing no parameters. Multi-Head LatentMoE has a total FLOPs equaling that of an MoE, discounting the linear projections. However, a naive implementation would store N_{h} sets of routing scores and expert activations, multiplying the HBM and IO cost by N_{h} . We address this challenge in Section 3.3 and Section 3.4 . 3.2 Head Parallel Multi-Head LatentMoE creates independent sub-tokens and MoE instances. This creates natural boundaries that divide expert weights and tokens. We propose Head Parallel, the distributed training method for Multi-Head LatentMoE. The key idea is to move all-to-all to before routing. Specifically, let P denote the number of GPUs, where P\leq N_{h} and N_{h} is divisible by P . Initially, each GPU holds a tensor of sub-tokens with shape (B,T,N_{h},d_{h}) . All-to-all redistributes this tensor so that each GPU receives all sub-tokens for its assigned heads, yielding shape (B,T,P,N_{h}/P,d_{h}) . After MoE computation, a reverse all-to-all sends the outputs back to the original GPUs. Head Parallel is not fundamentally bounded by N_{h} . It composes naturally with other parallelism strategies, such as Expert Parallel, to scale beyond N_{h} GPUs. The key insight is that HP provides a communication-efficient highway for distributing sub-tokens, which can serve as a building block within larger distributed training systems. Head Parallel streamlines token communication. First, each token is sent exactly once without duplication, so communication volume is O(1) relative to k . Second, each GPU sends and receives the same amount of data deterministically, so there is no need to communicate any metadata beforehand. This also avoids edge cases such as waiting for the slowest node or out-of-memory, when too many tokens route to a single GPU. We verify these advantages in Section 4.2 . Algorithm 1 IO-Aware Routing: Forward Pass 0: Sub-tokens \mathbf{X}\in\mathbb{R}^{B\times T\times N_{h}\times d_{h}} , router weights \mathbf{W}_{r}\in\mathbb{R}^{N_{h}\times d_{h}\times N_{e}} , load-balancing bias \mathbf{b}\in\mathbb{R}^{N_{h}\times N_{e}} , number of active experts k , block sizes N (tokens), M (experts). 0: Top- k scores \mathbf{S}_{\text{top}}\in\mathbb{R}^{B\times T\times N_{h}\times k} , top- k indices \mathbf{I}_{\text{top}}\in\mathbb{N}^{B\times T\times N_{h}\times k} . 1: Divide \mathbf{X} into \lceil T/N\rceil blocks of N tokens each. 2: for each (b,t_{\text{block}},h) in parallel do 3: Load \mathbf{X}_{\text{block}}\in\mathbb{R}^{N\times d_{h}} from HBM to SRAM. 4: Initialize accumulator \mathcal{A}\in\mathbb{N}^{N\times k} with 0 . 5: for e_{\text{block}}=0 to \lceil N_{e}/M\rceil-1 do 6: Load \mathbf{W}_{r,\text{block}}\in\mathbb{R}^{d_{h}\times M} and \mathbf{b}_{\text{block}}\in\mathbb{R}^{1\times M} from HBM to SRAM. 7: On chip, compute \mathbf{S}_{\text{block}}=\mathbf{X}_{\text{block}}\mathbf{W}_{r,\text{block}}+\mathbf{b}_{\text{block}}\in\mathbb{R}^{N\times M} . 8: On chip, pack scores and indices into 64-bit unsigned integers. 9: On chip, find top- k of \mathbf{S}_{\text{block}} along expert dimension. 10: On chip, merge with \mathcal{A} and retain the best k . 11: end for 12: On chip, unpack \mathcal{A} to scores \mathbf{S}^{\prime}_{\text{top}} and indices \mathbf{I}_{\text{top}} . 13: On chip, compute \mathbf{S}_{\text{top}}=\mathbf{S}^{\prime}_{\text{top}}-\mathbf{b}[\mathbf{I}_{\text{top}}] (remove bias). 14: Write \mathbf{S}_{\text{top}} and \mathbf{I}_{\text{top}} to HBM. 15: end for Algorithm 2 IO-Aware Routing: Backward Pass 0: Sub-tokens \mathbf{X}\in\mathbb{R}^{B\times T\times N_{h}\times d_{h}} , router weights \mathbf{W}_{r}\in\mathbb{R}^{N_{h}\times d_{h}\times N_{e}} , top- k indices \mathbf{I}_{\text{top}}\in\mathbb{N}^{B\times T\times N_{h}\times k} , gradient of top- k scores \mathrm{d}\mathbf{S}_{\text{top}}\in\mathbb{R}^{B\times T\times N_{h}\times k} , block size N (tokens). 0: Input gradient \mathrm{d}\mathbf{X}\in\mathbb{R}^{B\times T\times N_{h}\times d_{h}} , router weight gradient \mathrm{d}\mathbf{W}_{r}\in\mathbb{R}^{N_{h}\times d_{h}\times N_{e}} . 1: Initialize \mathrm{d}\mathbf{W}_{r}=\mathbf{0} in HBM. 2: Divide \mathbf{X} , \mathbf{I}_{\text{top}} , \mathrm{d}\mathbf{S}_{\text{top}} into \lceil T/N\rceil blocks of N tokens each. 3: for each (b,t_{\text{block}},h) in parallel do 4: Load \mathbf{X}_{\text{block}}\in\mathbb{R}^{N\times d_{h}} from HBM to SRAM. 5: Initialize \mathrm{d}\mathbf{X}_{\text{accum}}=\mathbf{0}\in\mathbb{R}^{N\times d_{h}} on chip. 6: for i=1 to k do 7: Load \mathbf{e}_{i}=\mathbf{I}_{\text{top}}[:,:,:,i]\in\mathbb{N}^{N} and \mathrm{d}\mathbf{s}_{i}=\mathrm{d}\mathbf{S}_{\text{top}}[:,:,:,i]\in\mathbb{R}^{N} from HBM to SRAM. 8: Load \mathbf{W}_{r}[:,\mathbf{e}_{i}]\in\mathbb{R}^{N\times d_{h}} from HBM to SRAM (gather by expert index). 9: On chip, compute \mathrm{d}\mathbf{X}_{\text{accum}}\mathrel{+}=\mathrm{d}\mathbf{s}_{i}\cdot\mathbf{W}_{r}[:,\mathbf{e}_{i}] . 10: Atomic add \mathbf{X}_{\text{block}}^{\top}\mathrm{d}\mathbf{s}_{i} to \mathrm{d}\mathbf{W}_{r}[:,\mathbf{e}_{i}] in HBM. 11: end for 12: Write \mathrm{d}\mathbf{X}_{\text{accum}} to \mathrm{d}\mathbf{X} in HBM. 13: end for 3.3 IO-Aware Routing Multi-Head LatentMoE projects each token into N_{h} sub-tokens. However, a naive router materializes the routing scores in HBM before selecting the top- k experts. With N_{h}=8 , the HBM and IO costs are 8 times higher. We propose IO-aware routing, inspired by FlashAttention ( 6 ) . The key observation is that a router only outputs the top- k indices and scores. Therefore, we can stream the top- k results in SRAM without materializing in HBM. The forward pass is given in Algorithm 1 . For each block of experts, we compute scores in SRAM and find the block’s local top- k results. The local results are merged with an accumulator to track the running top- k indices and scores, which are written to HBM by the end. It is worth mentioning two computational tricks. First, to support aux-free load balancing ( 29 ) , we add bias before top- k and subtract it afterward. As such, bias influences routing without affecting the final scores. Second, following 27 , we pack the score and index into a 64-bit unsigned integer to achieve arg-top- k in Triton. The backward pass is given in Algorithm 2 . Given \mathbf{S}=\mathbf{X}\mathbf{W} , the input gradient is \mathrm{d}\mathbf{X}=\mathrm{d}\mathbf{S}\cdot\mathbf{W}^{\top} and the weight gradient is \mathrm{d}\mathbf{W}=\mathbf{X}^{\top}\cdot\mathrm{d}\mathbf{S} . Gradients are computed only for the k selected experts per token, reducing time complexity from O(N_{e}) to O(k) , given that k<