Mixture of Lookup Experts
Shibo Jie, Yehui Tang, Kai Han, Yitong Li, Duyu Tang, Zhi-Hong Deng, Yunhe Wang · 2025
Read Mixture of Lookup Experts by Shibo Jie, Yehui Tang, Kai Han, Yitong Li, Duyu Tang, Zhi-Hong Deng, Yunhe Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15798.txt.
Introduction
Shibo Jie and co-authors redesign mixture-of-experts language models so their experts can live on cheap storage instead of pricey graphics memory, in a 2025 paper. In such models only a few expert sub-networks run per word, which keeps computation low, yet all experts must sit in fast memory because any might be called. Fetching them on demand slows everything down.
Their design, Mixture of Lookup Experts, sidesteps the problem. Each expert takes the word-embedding layer's output as its input, so after training its answers for every possible input can be precomputed into a lookup table. At run time the model simply retrieves the stored answer, with negligible transfer cost. Tests show speeds matching ordinary dense models and far beating expert-offloading designs, at the same compute and memory budgets, while quality stays level with standard mixture-of-experts models.
Catalog record
| CREATOR | Shibo Jie, Yehui Tang, Kai Han, Yitong Li, Duyu Tang, Zhi-Hong Deng, Yunhe Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15798
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mixture of Lookup Experts Shibo Jie, Yehui Tang, Kai Han, Yitong Li, Duyu Tang, Zhi-Hong Deng, Yunhe Wang Abstract Mixture-of-Experts (MoE) activates only a subset of experts during inference, allowing the model to maintain low inference FLOPs and latency even as the parameter count scales up. However, since MoE dynamically selects the experts, all the experts need to be loaded into VRAM. Their large parameter size still limits deployment, and offloading, which load experts into VRAM only when needed, significantly increase inference latency. To address this, we propose Mixture of Lookup Experts (MoLE), a new MoE architecture that is efficient in both communication and VRAM usage. In MoLE, the experts are Feed-Forward Networks (FFNs) during training, taking the output of the embedding layer as input. Before inference, these experts can be re-parameterized as lookup tables (LUTs) that retrieves expert outputs based on input ids, and offloaded to storage devices. Therefore, we do not need to perform expert computations during inference. Instead, we directly retrieve the expert’s computation results based on input ids and load them into VRAM, and thus the resulting communication overhead is negligible. Experiments show that, with the same FLOPs and VRAM usage, MoLE achieves inference speeds comparable to dense models and significantly faster than MoE with experts offloading, while maintaining performance on par with MoE. Code: https://github.com/JieShibo/MoLE . Figure 1: With the same 410M activated parameters, MoE outperforms the dense model in terms of performance, but it comes with significant VRAM usage. If experts are offloaded, inference latency will increase. Our MoLE maintains competitive performance without increasing the model’s VRAM usage or decoding latency. 1 Introduction Scaling laws indicate that, with sufficient data for training, the performance of large language models (LLMs) improves as the model size increases ( Kaplan et al. 2020 ) . However, larger LLMs also result in slower inference speeds, which can degrade the user experience. For this reason, the architecture of LLMs has increasingly focused on Mixture of Experts (MoE) ( Jiang et al. 2024 ; Dai et al. 2024 ) . MoE models use several Feed-Forward Networks (FFNs) as experts and employ a router to determine which subset of experts needs to be activated, rather than activating the entire model. This allows the model to maintain a large number of parameters while keeping the computational cost low. Although MoE reduces the computational cost, the number of parameters does not decrease. This means that the VRAM requirements during inference remain unaffordable. For example, although the Mixtral-8 \times 7B ( Jiang et al. 2024 ) model only has 13B parameters activated at a time, its total parameter count reaches 46B, making it impossible to load into a single 80GB GPU with FP16. Existing methods ( Eliseev & Mazur 2023 ; Xue et al. 2024 ; Shen et al. 2022 ) reduce VRAM usage by offloading experts to larger storage devices ( e.g. , CPU RAM, disk, or cloud storage), and loading the selected experts into VRAM at each inference step. However, there are two drawbacks to this approach: i) Since the selection of experts is dynamically determined by the router, we must load different experts into VRAM at each inference step. Frequent transfer of large numbers of parameters can significantly increase inference latency, as in Figure 1 . ii) Since different samples select different experts within a single step, loading only a subset of experts may not meet the needs of batched generation. To address the issues mentioned above, we propose Mixture of Lookup Experts (MoLE), a new LLM architecture. MoLE has different structures in training and inference. During training, MoLE is similar to MoE, with a router and several experts. However, unlike MoE, where experts take intermediate features as input, MoLE’s experts are fed with embedding tokens ( i.e. , the output of the embedding layer) instead. Additionally, MoLE allows all experts to be activated simultaneously. After training, MoLE is not directly used for inference but undergoes a series of re-parameterizations. Since the output of the embedding layer is fixed for specific input ids, the inputs to the experts have only a limited number of choices, equal to the model’s vocabulary size. Therefore, for each token in the embedding layer, we pre-compute the outputs corresponding to all experts, creating lookup tables (LUTs) that replaces the original experts. During inference, MoLE demonstrates several advantages: • Computation-free experts. The experts are re-parameterized from FFNs into LUTs, eliminating the need for any computation. Each expert only requires a single lookup operation. • Low VRAM overhead and communication latency. Although the size of the LUT is much larger than the model itself, it can be entirely offloaded to storage devices. During inference, since the output of each expert is the same number of tokens as the input, the communication required to load the lookup results into VRAM is negligible, thereby avoiding increased inference latency. • Batch-generation friendly. Traditional expert offloading methods introduce additional VRAM usage and communication latency during batched generation because different samples in a batch may select different experts. MoLE only transfers the pre-computed expert outputs, making its communication overhead still negligible even during batched generation. Through extensive experiments, we validated the effectiveness of MoLE at scales of 160M, 410M, and 1B parameters. As in Figure 1 , with equivalent computational cost and VRAM usage, MoLE significantly outperforms dense models while maintaining the same inference speed. Compared to MoE with expert offloading, MoLE achieves better performance and with significantly faster inference speed. 2 Related Work 2.1 Mixture-of-Experts The concept of MoE was initially introduced by Jacobs et al. 1991 ; Jordan & Jacobs 1994 and has been widely explored and developed through subsequent research ( Collobert et al. 2002 ; Rasmussen & Ghahramani 2001 ; Shahbaba & Neal 2009 ; Eigen et al. 2014 ; Theis & Bethge 2015 ; Deisenroth & Ng 2015 ; Aljundi et al. 2017 ; Shazeer et al. 2017 ) . MoE posits that different parts of the model, i.e. , the experts, focus on distinct tasks or encapsulate different kinds of knowledge. In this paradigm, only the experts relevant to a given input are activated, which allows the model to scale its capacity while keeping computational costs manageable and making full use of specialized knowledge across many experts. As the scale of LLMs increases, reducing computational overhead has become a key focus. This leads to its application in transformer-based LLMs ( Lepikhin et al. 2021 ) , making MoE a widely used architecture. Recently, a series of industrial-scale large language models have been released, including Mixtral ( Jiang et al. 2024 ) and DeepSeek-MoE ( Dai et al. 2024 ) . Some works also aim to improve the efficiency of MoE ( Jin et al. 2025 ) or increase expert capacity ( Yan et al. 2025 ) by modifying the model architecture. 2.2 Expert Offloading Offloading techniques typically transfer a portion of the model parameters to CPU RAM or disk when GPU memory is insufficient. However, current mainstream offloading frameworks, such as Zero-Infinity ( Rajbhandari et al. 2021 ) , are designed for dense LLMs and load model parameters layer by layer on-demand. This approach overlooks the sparse activation property of MoE models, leading to the unnecessary loading