MoPEQ: Mixture of Mixed Precision Quantized Experts

Krishna Teja Chitty-Venkata, Jie Ye, Murali Emani · 2025

Read MoPEQ: Mixture of Mixed Precision Quantized Experts by Krishna Teja Chitty-Venkata, Jie Ye, Murali Emani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02512.txt.

Introduction

Krishna Teja Chitty-Venkata, Jie Ye, and Murali Emani present a 2025 machine learning systems paper on shrinking mixture-of-experts models, which split work among many expert subnetworks. Such models strain memory when deployed, especially in combined vision-language systems.

Their MoPEQ method assigns a different bit width, the number size used in storage, to each expert after training. Sensitivity is judged by a Hessian trace approximation, a curvature-based score, rather than by how often an expert fires. Similar experts are clustered to protect accuracy. Tests on vision-language benchmarks with several DeepSeek and MolmoE models kept accuracy competitive while cutting memory sharply against uniform precision. A broader study varies two, three, and four bit settings across layers and models.

Catalog record

CREATORKrishna Teja Chitty-Venkata, Jie Ye, Murali Emani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02512

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MoPEQ: Mixture of Mixed Precision Quantized Experts Krishna Teja Chitty-Venkata, Jie Ye, Murali Emani Abstract Large Language and Vision Models using a Mixture-of-Experts (MoE) architecture pose significant challenges for deployment due to their computational and memory demands. Mixed Precision Quantization assigns different precisions to different layers of an LLM/VLM based on layer sensitivity and importance within the model. In this work, we propose a Post Training Quantization algorithm, MoPEQ, that assigns optimal bit width to each expert. Our method balances accuracy and model size by analyzing each expert’s sensitivity using Hessian trace approximation instead of relying on the activation frequency of the expert. This per-expert granularity approach clusters similar experts to maintain model performance while reducing memory requirements. The experimental results on VLMEvalKit benchmark datasets using State-of-the-art VLMs Deepseek-VL2 -tiny, -small, -base, and MolmoE models demonstrate that our mixed precision quantized MoEs achieve competitive accuracy with substantial improvements in memory footprint compared to uniform-precision baseline methods. We perform a comprehensive study to analyze the impact of expert activation frequency and sensitivity using Hessian trace approximation at both layer-wise and model-wide expert precision allocation of 2, 3, and 4 bits to provide a thorough understanding of mixed precision quantization of VLM-MoEs. The code is available here . 1 Introduction Large Language Models (LLMs), such as GPT [ 3 ] , PaLM [ 6 ] , Deepseek-V3 [ 28 ] and LLaMA [ 37 ] , are based on transformer architectures trained on vast text data to learn and generate human-like language. These networks achieve remarkable performance in tasks such as text completion and translation using hundreds of billions of parameters and attention mechanisms. Vision-Language Models (VLMs) [ 44 ] integrate visual and text understanding to tackle multimodal tasks such as image-text retrieval, visual question answering (VQA), and image captioning. These models employ cross-modal transformers to align visual features with text representations, enabling tasks requiring reasoning over images/videos and text. Figure 1 : MoPEQ: Mixed Precision Quantization of MoE A Mixture of Expert (MoE) [ 4 , 29 , 35 ] , such as Deepseek-MoE [ 7 ] , is a type of network where multiple specialized subnetworks (or experts in general) exist within the same network. The experts are activated through a learned gating network. This kind of architecture allows the execution of complex tasks by dividing them among smaller, specialized subnetworks. Unlike traditional dense models, MoE experts are not pre-defined for specific domains but instead develop their specializations organically during training. The gating network plays a key role by activating only the most suitable experts based on the input tokens. This type of approach enhances computational efficiency and improves task performance by leveraging the strengths of each expert. LLM Quantization [ 46 ] is an efficient method to address the storage challenges imposed by the immense computational and memory demands of LLMs and VLMs. The model size can be decreased by reducing the numerical precision of parameters, accelerating inference while maintaining model accuracy [ 38 ] . There are two approaches to quantize a pretrained LLM, which are as follows: (i) Post Training Quantization (PTQ) is a method to reduce precision after training the model, and (ii) Quantization Aware Training (QAT) simulates the low-precision effects during training to mitigate accuracy loss, although requiring an additional cost. PTQ methods (Eg: GPTQ [ 15 ] and AWQ [ 26 ] ) and QAT methods (Eg: QAT-LLM [ 30 ] ) demonstrate robust results, compressing models by up to 4 \times with minimal accuracy loss. Mixed precision quantization can optimize LLMs by assigning different precisions to different layers to preserve accuracy. This approach can balance hardware efficiency and accuracy by assigning higher precision to critical parts of the model while using lower precision for less sensitive components. Mixed precision quantization is particularly well-suited for MoE models due to their unique architecture, where only a subset of experts is activated for any given input. This selective activation enables targeted compression strategies to reduce the size of MoE models as uniform quantization can lead to suboptimal compression. Challenges and limitations of State-of-the-art Methods: The problem of MoE mixed precision quantization for Vision Language models remains underexplored, although there are a few works in the LLM space. A common approach in the literature to quantize experts in MoE models is to allocate precision based on the expert activation frequency [ 18 , 24 ] . Specifically, experts that are activated more frequently are assigned higher bit widths. However, this method does not account for each expert’s sensitivity to quantization. An expert with low activation frequency is not necessarily inactive throughout the inference. In scenarios where such experts are used for generation, quantization-induced errors can propagate throughout the network, especially if the expert is highly sensitive to quantization. Additionally, models like DeepSeek-V2 [ 39 ] incorporate load balancing loss during training to ensure uniform utilization of all experts, making it challenging to differentiate between the most and least important experts. Computing activation frequency also requires a calibration dataset, which may not always be representative of all the diverse and varying tasks. Calibrating with limited data samples to capture activation frequency can tune the quantization to a specific dataset but may limit its generalizability across diverse tasks. MoPEQ: To address the challenges of MoE quantization, we propose MoPEQ, an efficient algorithm for allocating different bit widths to experts (illustrated in Figure 1 ). Our approach leverages Hessian trace approximation to estimate expert sensitivity without relying on a dataset. Unlike previous layer-wise assignment methods that focus only on mixed precision within a layer, we allocate higher bit width to layer with high sensitivity. We construct a mixed-precision search space and employ an expert clustering method to group similar experts based on their relative importance. Contributions The main contributions of our paper are: (1) We introduce MoPEQ , a mixed-precision assignment algorithm for allocating varying bit widths to different experts in MoE layers. To the best of our knowledge, there are no published works on mixed-precision quantization for VLM-MoEs. We investigate the impact of mixed precision expert quantization on vision and language token scenarios. (2) We compare our expert mixed precision assignment method based on expert sensitivity with the activation frequency approach [ 24 ] and layer-wise expert precision assignment [ 18 ] . We show that the sensitivity of each expert plays a bigger role than activation frequency. We also demonstrate that our sensitivity-based approach can distinguish the importance of experts and reduce the model size where activation frequency is uniform across layers. In this work, we limit our mixed precision scope only to experts in MoE layers and other layers are quantized uniformly. (3) We evaluate our quantization approach on four state-of-the-art VLMs: MolmoE-1B [ 11 ] and DeepSeek-VL2 (tiny, small, and base variants) [ 39 ] , using multiple VLMEvalKit tasks [ 14 ] . Our results demonstrate a model size reduction of approximately 1.5\times while maintaining accuracy within 5% across seven tasks. 2 Background 2.1 Mixture of Expert (MoE) A Mixture of Expert architecture redesigns the traditional dense network by decomposing monolithic layers into specialized subnetworks called experts E_{1},E_{2},\ldots,E_{N} . These experts are chosen per token based on a trainable gating mechanism G(x) . For an

Read the complete text