Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees

Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr, Kaoutar El Maghraoui, Liu Liu, Meng Wang · 2026

Read Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees by Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr, Kaoutar El Maghraoui, Liu Liu, Meng Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02633.txt.

Introduction

Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey Burr, and colleagues address a hardware problem for very large language models in 2026. Mixture-of-Experts models use only a few expert parts per input, yet still hold huge numbers of weights. Analog in-memory chips could run them with far less energy, but chip noise hurts accuracy.

Their fix splits the work instead of retraining the model. Experts shown to be noise-sensitive, plus attention layers, run on normal digital hardware. The rest run on the analog chips. Tests on two large expert models across several benchmarks kept accuracy steady despite analog flaws. The split makes low-energy hardware usable for models too big to retrain.

Catalog record

CREATORMohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr, Kaoutar El Maghraoui, Liu Liu, Meng Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54722
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.02633

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr, Kaoutar El Maghraoui, Liu Liu, Meng Wang Abstract Sparse Mixture-of-Experts (MoE) models enable efficient scalability by activating only a small subset of experts per input, yet their massive parameter counts lead to substantial memory and energy inefficiency during inference. Analog in-memory computing (AIMC) offers a promising solution by eliminating frequent data movement between memory and compute units. However, mitigating hardware nonidealities of AIMC typically requires noise-aware retraining, which is infeasible for large MoE models. In this paper, we propose a retraining-free heterogeneous computation framework in which noise-sensitive experts, which are provably identifiable by their maximum neuron norm , are computed digitally while the majority of the experts are executed on AIMC hardware. We further assign densely activated modules, such as attention layers, to digital computation due to their high noise sensitivity despite comprising a small fraction of parameters. Extensive experiments on large MoE language models, including DeepSeekMoE and OLMoE, across multiple benchmark tasks validate the robustness of our approach in maintaining accuracy under analog nonidealities. 1 Introduction The sparse computation of Mixture-of-Experts (MoE) models allows efficient scaling of large language and vision models without the proportional increase of training compute ( 39 ; 36 ; 15 ; 11 ) . For each input token, the model activates only a small subset of group of neurons in the feed forward network (FFN) modules of Transformer. Each of the group is referred to as an expert . Due to their excellent scalability, modern MoE-based large language models (LLMs) have grown to disproportionately large parameter sizes ( 25 ; 20 ; 44 ) , which in turn require substantial memory to operate and result in significant energy inefficiency while computing in digital accelerators ( 17 ; 10 ; 24 ; 16 ) . The analog in-memory computing (AIMC) provides a promising solution to the inefficiencies of MoE ( 7 ) . AIMC allows to compute the matrix-vector multiplication (MVM) inside the non-volatile memory (NVM) devices, thus eliminating the need for frequent movement of large expert parameters between the memory and compute units, which is the major source of energy inefficiency ( 8 ; 26 ; 38 ; 18 ) . However, MVM operations on AIMC hardware are inherently approximate due to various nonidealities, such as noise introduced by digital-to-analog and analog-to-digital conversions (DAC and ADC) and inaccuracies in programming weights onto NVM devices ( 22 ) . These nonidealities can lead to substantial degradation in model performance. A common strategy to alleviate this degradation is noise-aware retraining, which improves robustness to AIMC-induced noise ( 35 ; 5 ) . However, the large parameter counts of modern MoE models renders such retraining approaches impractical for AIMC deployment. For a robust retraining-free deployment of large MoE models in AIMC requires a heterogeneous computing approach, where the most sensitive components (e.g., noise-sensitive experts) are computed in digital accelerators, and the rest of the model components are computed in AIMC devices. However, a systematic approach is required to identify the noise-sensitive model components for the pareto-optimal heterogeneous deployment. This raises a fundamental questions: How to identify the noise-sensitive model components for the heterogeneous Digital-AIMC deployment of MoE? Figure 1 : a. In heterogeneous computing of MoE, the dense modules and noise-sensitive expert modules are computed in a digital accelerator while rest of the experts are computed in an analog accelerator. b. A schematic of digital and analog accelerator. c. The analog accelerator is comprised of non-volatile memory (NVM) tiles. Weights are programmed to the crossbar array of the tile. In this paper, we address the question both theoretically and empirically. Our theoretical analysis reveals that the experts composed of neurons with large \ell_{2} -norms are more susceptible to weight-programming noise in AIMC. We therefore propose to compute these experts in digital, while computing rest of the experts in AIMC devices. We further propose to compute the dense model components (e.g., the multi-head self-attention modules, the language modeling head, and the shared experts (if any) of MoE-LLMs) in digital due to their high per-parameter sensitivity to analog noises. We summarize our main contributions as follows: 1. The first analog noise-sensitivity analysis of large MoE models. To the best of our knowledge, this paper provides the first systematic sensitivity analysis of different model components in large MoE models under the two major sources of AIMC nonidealities: the DAC-ADC noise, and the weight-programming noise. Our analysis reveals that a subset of experts exhibits pronounced performance degradation as weight-programming noise increases, suggesting the computation of these experts in digital. Furthermore, we show that dense model components in MoE—despite accounting for only a small portion of the total parameters—are highly sensitive to both DAC–ADC and weight-programming noise due to their involvement in processing all input tokens. 2. A theoretically grounded metric—the expert maximum neuron norm score —for digital expert selection. We theoretically prove that, the experts specializing in frequently appearing important tokens in data contain neurons with large \ell_{2} norms, hence demonstrate high sensitivity to weight-programming noise. We therefore propose to compute the experts with large maximum neuron norm scores —defined as the product of the maximum neuron \ell_{2} norms across all linear projection layers within an expert—in digital. 3. Empirical validation of the proposed heterogeneous computing approach on large MoE models. We empirically evaluate the proposed heterogeneous computing framework on large-scale MoE models, including DeepSeekMoE ( 14 ) (16B parameters) and OLMoE ( 34 ) (7B parameters), across a diverse set of benchmark LLM tasks. The results demonstrate that our maximum neuron norm–based digital expert selection consistently outperforms prior expert selection strategies proposed in the MoE literature, demonstrating robustness in maintaining accuracy under imperfect analog computation. 1.1 Related Works Analog in-memory computing of large models. The evaluation of in-memory computing performance is largely limited to smaller size models of different architectures such as CNNs ( 26 ; 19 ) , RNNs ( 35 ) , LSTMs ( 42 ) , and encoder only Transformers ( 41 ) . ( 46 ) minimize DAC-ADC noise for LLMs through activation-shifting and weight-reshaping. ( 5 ) conduct large scale noise-aware retraining for foundation LLMs. ( 22 ) conduct sensitivity analysis of LLMs on various analog noise sources and proposes a training-free rescaling method to mitigate the noise. In all of the cases, the model sizes (1-3B) are well below typical MoE parameter ranges. Efficient inference of Mixture-of-Experts. Due to the large memory requirements of MoE, inference cost reduction has drawn much attention recently ( 28 ; 33 ; 10 ; 17 ; 23 ) . Most of the works focus on expert pruning and quantization to mitigate inference cost. ( 7 ) demonstrate improved scalability of MoE in 3D AIMC architecture compared to dense models. Theoretical generalization guarantees of neural networks. Due to high non-convexity, the theoretical generalization analysis of neural networks is mostly limited to two-layers ( 11 ; 4 ; 32 ) . Feature learning dynamics is widely used to gain insights about different training and inference properties ( 31 ; 30 ) . Most works require simplified data model to facilitate analysis ( 3 ; 27 ; 40 ) . 2

Read the complete text