LExI: L ayer-Adaptive Active Ex perts for Efficient MoE Model I nference
Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, Venkatram Vishwanath · 2025
Read LExI: L ayer-Adaptive Active Ex perts for Efficient MoE Model I nference by Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, Venkatram Vishwanath (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02753.txt.
Introduction
Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, and Venkatram Vishwanath wrote a 2025 machine learning systems paper on speeding up mixture-of-experts model inference. Such models activate a few experts per token, but standard designs use the same count in every layer.
The authors first show that pruning experts mainly saves memory, not run time, on optimised GPU frameworks. Their LExI method instead chooses how many experts each layer should activate, using only model weights and no extra data. Layer importance scores drive the assignment, cutting redundant computation. On language and vision benchmarks, LExI beat pruning methods on efficiency with negligible accuracy loss. In one test, a tuned model matched a pruned rival's speed while scoring clearly higher accuracy.
Catalog record
| CREATOR | Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, Venkatram Vishwanath |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-02753
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LExI: L ayer-Adaptive Active Ex perts for Efficient MoE Model I nference Krishna Teja Chitty-Venkata, Sandeep Madireddy, Murali Emani, Venkatram Vishwanath Abstract Mixture-of-Experts (MoE) models scale efficiently by activating only a subset of experts per token, offering a computationally sparse alternative to dense architectures. While prior post-training optimizations, such as inter- and intra-expert pruning, reduce memory usage but provide limited gains in inference-time compute efficiency. Moreover, existing MoE architectures typically activate a fixed number of experts uniformly across all layers, resulting in redundant computation and suboptimal performance. In this work, we first demonstrate that MoE pruning strategies improve only the memory footprint but do not significantly improve inference performance on GPU using optimized frameworks such as vLLM. To address this, we introduce LExI , a data-free optimization technique that determines the optimal number of active experts per layer in a pretrained MoE model. LExI leverages only the model’s weights to estimate the relative importance of each layer and adaptively assigns the number of active experts accordingly per layer. Experiments on state-of-the-art language and vision MoE benchmarks demonstrate that LExI significantly outperforms traditional MoE pruning approaches in terms of inference efficiency with negligible accuracy loss. For example, using LExI, Qwen1.5-MoE achieves the same throughput on Nvidia H100 GPU with 10% better accuracy than traditional expert pruning. 1 Introduction Figure 1 : Overview of MoE Optimization Methods. (a) Baseline Trained Model (b) Unstructured Expert Pruning (SparseGPT ( Frantar and Alistarh 2023 ), Wanda ( Sun et al. 2023 ), MoE-Pruner ( Xie et al. 2024 )) (c) Inter Expert Pruning (NAEE ( Lu et al. 2024 )) (d) Dynamic Expert Skipping (NAEE ( Lu et al. 2024 )) (e) Intra Expert Pruning (MoE-I 2 ( Yang et al. 2024 )) (f) LExI : Static Varying Active Experts (Topk) Per Layer (Ours) Large Language Models (LLMs) and Vision Language Models (VLMs) have achieved remarkable performance through model scaling, but require tremendous compute and memory resources. Mixture-of-Experts (MoE) models have emerged as a promising approach to increase model capacity without a proportional rise in inference cost. In an MoE, multiple expert subnetworks are trained, and a sparse gating or router network activates only a small subset of experts ( k experts) per input token. This sparse computation allows MoEs to outperform dense models with the same number of active model parameters. Prominent examples of MoE-based LLM include Mixtral ( Jiang et al. 2024 ), Qwen1.5-MoE-A2.7B ( Team 2024 ), OLMoE ( Muennighoff et al. 2024 ), and Vision Language Models (VLM) such as MolmoE ( Deitke et al. 2024 ) and DeepSeek-VL2 ( Wu et al. 2024 ). One commonly used post-training optimization strategy for MoE models is expert pruning, which removes redundant experts. Recent methods such as NAEE ( Lu et al. 2024 ), MoE-Pruner ( Xie et al. 2024 ) EEP ( Liu et al. 2024b ) and MoE-I 2 ( Yang et al. 2024 ) introduce various pruning strategies. For example, NAEE identifies and removes entire experts from a pretrained MoE model, while MoE-I 2 prunes the inner dimensions within each expert’s MLP. Although these methods reduce the memory footprint, our performance evaluation across several state-of-the-art MoE models using the widely adopted vLLM inference framework reveals a critical limitation: pruning does not consistently translate into faster inference , and in some cases, it even degrades performance. This degradation is primarily due to the sparse structure of the MoE models itself. In expert pruning, the input token still needs to be routed to the same number of top- k experts, as determined by the router. While some experts are pruned, the remaining ones must process a disproportionately larger number of tokens, increasing their computational load. In the batched inference scenario, this load imbalance can lead to longer processing time per expert, thereby increasing overall latency. While aggressive expert pruning levels can yield noticeable speedups, they typically result in significant accuracy degradation, making them impractical. Moreover, these expert pruning approaches usually rely on training data for pruning experts. Current MoE architectures employ a fixed top- k routing mechanism, where the same number of experts are activated for each input token across all layers. This static design is suboptimal as different layers may require varying levels of expert capacity depending on computational needs ( Guo et al. 2024 ). Beyond computation, inter-GPU communication overhead also becomes a significant performance bottleneck in MoEs. Increasing the number of active experts per token increases the volume of communication operations such as all-reduce and broadcast, further adding to the inference cost. To address some of these limitations, NAEE ( Lu et al. 2024 ) proposed a token-aware dynamic expert skipping strategy, which selectively skips an expert during inference. However, this strategy is highly tailored to the dataset and cannot work beyond top- k =2. In summary, existing MoE optimization strategies often rely on calibration sets for pruning or routing adjustments, making them unsuitable in deployment settings where access to data or retraining is infeasible. In addition, the dataset-driven solutions will make pruned models more optimized to that calibration dataset, potentially degrading the performance in unseen settings. Motivated by these insights, we propose LExI , a novel post-training layer-adaptive active expert allocation mechanism that determines the optimal number of active experts per layer without depending on any dataset. Our method relies on the key observation that not all layers contribute equally to the final model performance and that expert redundancy varies significantly across depth. This raises a fundamental question: Can we reduce the number of active experts per layer irrespective of the input token without sacrificing accuracy? In particular, is it possible to statically assign different top- k values to each layer, so that every layer uses just enough experts to retain its contribution, while improving overall inference efficiency? By making layer-adaptive expert allocation decisions, LExI reduces computational overhead across the model irrespective of the input token, offering a more efficient alternative to the traditional fixed top-K routing. Our experiments show that LExI outperforms existing expert pruning techniques in both task performance and runtime efficiency. By reducing the average number of activated experts per layer, LExI reduces latency and memory bandwidth usage while maintaining competitive task accuracy across diverse tasks. Contributions. Our key contributions are as follows: • We introduce LExI , a novel dataset-free optimization technique for static active expert assignment in pretrained MoE models. LExI is simple to implement and serves as an efficient, plug-and-play solution for inference across various frameworks. • We propose a data-free profiling strategy to estimate the sensitivity of each expert using only model expert weights. LExI combines this one-time profiling with evolutionary search to determine the optimal active experts layer in a computationally efficient manner. • Unlike prior methods that demonstrate improvements on a narrow set of MoE models (Mixtral-8x7B), our approach generalizes across multiple state-of-the-art MoE architectures in both language and vision domains. • We empirically show that expert pruning in MoE models does not significantly improve inference performance, and in some cases, can degrade it due to architectural sparsity and load imbalance. Our method provides a viable alternative to pruning, improving both accuracy and hardware efficiency without requiring retraining or access to calibration data. 2 Background and Related Work Mixture