BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference
Zewen Jin, Shengnan Wang, Jiaan Zhu, Hongrui Zhan, Youhui Bai, Lin Zhang, Zhenyu Ming, Cheng Li · 2025
Read BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference by Zewen Jin, Shengnan Wang, Jiaan Zhu, Hongrui Zhan, Youhui Bai, Lin Zhang, Zhenyu Ming, Cheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-16927.txt.
Introduction
Zewen Jin and colleagues restructure mixture-of-experts language models so their experts talk less and compute more. Such models route each token to a few specialist sub-networks, scaling cheaply. Fine-grained versions split experts smaller and activate more of them. The all-to-all chatter between devices then becomes the bottleneck, always at full model width.
Their design, BigMac, reorders the work: shrink first, communicate twice at low width, then grow back. Projections at each expert's door cut the message size. The small experts are redesigned to stay expressive under the new flow. Experiments match or beat fine-grained rivals with the same expert count, at lower communication cost. Training and inference both speed up. The paper targets the hidden tax that wiring, not math, imposes on big models.
Catalog record
| CREATOR | Zewen Jin, Shengnan Wang, Jiaan Zhu, Hongrui Zhan, Youhui Bai, Lin Zhang, Zhenyu Ming, Cheng Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-16927
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BigMac: A Communication-Efficient Mixture-of-Experts Model Structure for Fast Training and Inference Zewen Jin, Shengnan Wang, Jiaan Zhu, Hongrui Zhan, Youhui Bai, Lin Zhang, Zhenyu Ming, Cheng Li Abstract The Mixture-of-Experts (MoE) structure scales the Transformer-based large language models (LLMs) and improves their performance with only the sub-linear increase in computation resources. Recently, a fine-grained DeepSeekMoE structure is proposed, which can further improve the computing efficiency of MoE without performance degradation. However, the All-to-All communication introduced by MoE has become a bottleneck, especially for the fine-grained structure, which typically involves and activates more experts, hence contributing to heavier communication overhead. In this paper, we propose a novel MoE structure named BigMac, which is also fine-grained but with high communication efficiency. The innovation of BigMac is mainly due to that we abandon the c ommunicate- d escend- a scend- c ommunicate (CDAC) manner used by fine-grained MoE, which leads to the All-to-All communication always taking place at the highest dimension. Instead, BigMac designs an efficient d escend- c ommunicate- c ommunicate- a scend (DCCA) manner. Specifically, we add a descending and ascending projection at the entrance and exit of the expert, respectively, which enables the communication to perform at a very low dimension. Furthermore, to adapt to DCCA, we re-design the structure of small experts, ensuring that the expert in BigMac has enough complexity to address tokens. Experimental results show that BigMac achieves comparable or even better model quality than fine-grained MoEs with the same number of experts and a similar number of total parameters. Equally importantly, BigMac reduces the end-to-end latency by up to 3.09 × \times for training and increases the throughput by up to 3.11 × \times for inference on state-of-the-art AI computing frameworks including Megatron, Tutel, and DeepSpeed-Inference. Introduction Increasing the size of Transformer-based large language models (LLMs) can continuously improve downstream application performance. Such a phenomenon, known as the scaling law, has been demonstrated by the auto-regressive dense models such as GPT-series ( OpenAI et al. 2024 ) and Llama-series ( Dubey et al. 2024 ) . However, this comes at the price of higher computing complexity and more resource consumption. Fortunately, the Mixture-of-Experts (MoE) technique, capable of expanding the model size tens or even hundreds of times without significantly increasing the computation, is widely used in various emerging huge models, such as GShard ( Lepikhin et al. 2020 ) , GLaM ( Du et al. 2022 ) , Switch Transformer ( Fedus, Zoph, and Shazeer 2022 ) , and Mixtral ( Jiang et al. 2024 ) , each of which consists of hundreds of billion parameters or even beyond. Figure 1: Convergence result comparison of MoE models with three structures. GPT-Fine-Grained takes 38.9 hours to reach the target perplexity of 13.69, while GPT-BigMac spends only 22.8 hours (1.7 \times faster). GPT-Vanilla fails to converge to the target perplexity under time budget. Recently, DeepSeekMoE ( Dai et al. 2024 ) , a fine-grained and more parameter-efficient MoE structure, has been proposed. Compared to conventional MoE models, for the same model size, DeepSeekMoE has significantly more experts per MoE layer and fewer parameters per expert. It was demonstrated that such a new structure can achieve comparable or even better results than conventional MoE models with much less time complexity and hence it is adopted by many later released models, such as Qwen2 ( Yang et al. 2024 ) and DeepSeek-v2 ( DeepSeek-AI et al. 2024 ) . Figure 2: The MoE layers of different structures. Here, N represents the number of experts in the Vanilla MoE model and ACT represents the activation function like ReLU. W_{\downarrow} and W_{\uparrow} represent the descending and ascending projection matrix of an expert, respectively. W_{\downarrow}^{\prime} and W_{\uparrow}^{\prime} represent the projection matrices introduced in BigMac. However, the MoE faces a serious All-to-All communication bottleneck during both training and inference. This is mainly because of the underlying expert parallelism (EP) strategy, which assigns experts to several hardware accelerators to avoid out-of-memory errors or improve efficiency ( Fedus, Zoph, and Shazeer 2022 ) . As a result, for each MoE layer, two All-to-All communication steps are introduced for dispatching tokens to their best-fit experts, which may be stored in the other accelerators and then gathering the results back before proceeding to the next layer. Recent studies have already found that the All-to-All communication is a key issue leading to the low efficiency of MoE model training and inference ( Hu et al. 2024 ) . Even worse, this bottleneck will be more pronounced in the fine-grained MoE structure, which generally requires to activate more experts to ensure performance. In this paper, we propose a novel efficient MoE structure, named BigMac, which is also fine-grained but can greatly reduce the All-to-All communication overhead. Note that existing fine-grained MoE models adopt the c ommunicate- d escend- a scend- c ommunicate (CDAC) manner, performing the All-to-All at a high dimension, leading to a heavy communication overhead. In contrast, BigMac designs an efficient d escend- c ommunicate- c ommunicate- a scend (DCCA) manner, which is capable of performing the All-to-All communication at a very low dimension. Furthermore, to adapt to DCCA, we further re-design the structure of the small expert in BigMac, ensuring the complexity of each single expert for the overall performance of the whole model. Specifically, each expert in BigMac is composed of an ascending projection and a descending projection with an activation in between, which is the opposite of fine-grained MoE models. Briefly, we make the following contributions: • We propose a novel MoE structure named BigMac, which can greatly improve the efficiency of MoE models for both training and inference. In addition, BigMac no longer suffers from problems such as the limited expert capacity and limited top\_k , which are common restrictions in the existing MoE structures. • We design a d escend- c ommunicate- c ommunicate- a scend (DCCA) strategy to ensure that the communication is always executed at the very low dimension, hence the communication overhead is greatly reduced. To guarantee the computing efficiency, BigMac adopts the idea of fine-grained MoE models, namely, each MoE layer is composed of a large number of small experts, while the structure of each small expert is re-designed to adapt to the DCCA strategy. • We pre-train the MoE models with different MoE structures and show that BigMac converges faster than the other structures (shown in Figure 1 ). The results on multiple downstream tasks show that BigMac can achieve comparable or better performance against other baselines using the same amount of resources. Moreover, evaluations on state-of-the-art distributed training / inference frameworks, including Megatron, Tutel, and DeepSpeed-Inference, show that BigMac can significantly mitigate the communication overhead, reducing the end-to-end latency by up to 3.09 \times for training and increasing the throughput by up to 3.11 \times for inference. Related Work and Motivation Fine-grained MoE. Starting from GShard ( Lepikhin et al. 2020 ) , the Mixture-of-Experts (MoE) technology has been applied to the Transformer architecture, allowing for a significant increase in the number of parameters with only a sub-linear increase in computational resources. As illustrated in Figure 2 a, the dense Feed-Forward Network (FFN) modules in Transformer are replaced with MoE sub-layers, each consisting of multiple parallel experts.