ButterflyMoE: Sub-Linear Ternary Experts via Structured Butterfly Orbits

Aryan Karmore · 2026

Read ButterflyMoE: Sub-Linear Ternary Experts via Structured Butterfly Orbits by Aryan Karmore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13563.txt.

Introduction

Aryan Karmore packs mixture of experts nets into far less memory by a butterfly plan for ternary experts. A mixture of experts net holds many expert weight sets. In plain form each set is stored in full, so memory grows as the count times the square of the width. Edge gear cannot hold that growth.

Old squeeze tools cut the per set cost but leave the growth form. Karmore's experts share a butterfly orbit plan, a set of fixed swap paths. Expert weights are ternary, in three states. Shared paths plus three state weights cut the growth to sub linear. For edge AI teams, that form may let big expert nets run on small gear.

Catalog record

CREATORAryan Karmore
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13563

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ButterflyMoE: Sub-Linear Ternary Experts via Structured Butterfly Orbits Aryan Karmore Abstract Linear memory scaling stores N N independent expert weight matrices requiring 𝒪 ⁡ ( N ⋅ d 2 ) \mathcal{O}(N\cdot d^{2}) memory, which exceeds edge devices memory budget. Current compression methods like quantization, pruning and low-rank factorization reduce constant factors but leave the scaling bottleneck unresolved. We introduce ButterflyMoE, a method that treats experts not as independent weight matrices but as geometric reorientations of a unified shared quantized substrate. Diversity among experts arises from viewing different angles of shared capacity, not from redundant storage. By applying learned rotations to a shared ternary prototype, each expert yields 𝒪 ⁡ ( d 2 + N ⋅ d ​ log ⁡ d ) \mathcal{O}(d^{2}+N\cdot d\log d) memory—sub-linear in the number of experts. The key insight: training these rotations with quantization reduces activation outliers and stabilizes extreme low bit training, where static methods collapse. Across language modeling benchmarks, ButterflyMoE achieves 150 × \times memory reduction at 256 experts with negligible accuracy loss. ButterflyMoE allows multiple experts to fit on edge-constrained devices showing that geometric parameterization breaks linear scaling. 1 Introduction A (Mixture of Expert) MoE layer with 64 experts having dimension d=512 requires 256 MB of memory, exceeding edge device limits. This is the result of an architectural assumption that each expert requires separate parameters. Current compression methods retain linear memory scaling. Quantization (QMoE [ 9 ] , MoQE [ 15 ] ) reduces bit-width and achieves 4–5× compression but retains O(N\cdot d^{2}) memory growth.Even 2-bit quantization requires 128 MB per layer at 256 experts. The structural question of whether N experts need O(N) separate parameter sets is not addressed. In light of recent research on linear model connection, we propose that rather than being stored separately, experts can be formed from a shared prototype through straightforward transformations. This arrangement introduces sub-linear memory scaling. ButterflyMoE parameterizes N experts as learned rotations of a single ternary-quantized substrate. Each expert W_{i} is defined as: W_{i}=\mathcal{B}(\phi_{i})\cdot W_{\text{base}}\cdot\mathcal{B}(\theta_{i})^{T}, where W_{\text{base}}\in\{-1,0,+1\}^{d\times d} is a shared 1.58-bit weight matrix and \mathcal{B}(\phi_{i}),\mathcal{B}(\theta_{i}) are expert-specific Butterfly matrices with O(d\log d) parameters. Experts are never explicitly materialized. Inference proceeds by applying a rotation and a ternary matrix multiply. This yields O(d^{2}+N\cdot d\log d) memory. This structural change has three benefits:(i) Memory compression improves with expert count achieving a 150x compression at 256 experts, (ii) Per-expert input rotations suppress activation outliers,(iii) A ternary substrate with rotations reduce quantization error by 97% relative to post-training quantization. ButterflyMoE equals dense accuracy on multi-domain language modeling while enabling deployments that were previously unattainable. On a Jetson Nano, the 64 expert model uses 1.9 MB of memory, compared to 256 MB for the baseline, and trains stably without recovery stages. Contributions • We introduce ButterflyMoE, combining ternary quantization with learned Butterfly rotations to achieve O(d^{2}+N\cdot d\log d) memory complexity. • We demonstrate 150 \times compression at 256 experts with competitive accuracy, enabling massively parallel edge deployment and up to 99.5% memory bandwidth energy reduction under standard DRAM energy models. 2 Literature Review Mixture of Experts (MoE) models face a memory bottleneck during edge deployment. In standard MoE architectures, the memory usage grows linearly with the number of experts. A model with 64 experts and d=512 must store 64 independent 512 \times 512 weight matrices requiring 256 MB, exceeding the capacity of devices like Jetson Nano. Prior Work is organized across three axes: • Memory Compression Techniques • Quantization Aware Training • Structured Parameterization None of the current methods acheives sub-linear memory scaling in the number of experts while maintaining competetive accuracy. 2.1 MoE Memory Compression Existing compression methods remain limited due to linear expert scaling. Recent work has explored the following approaches: • Basic decomposition methods represent experts as linear combinations of shared basis. MoBE [ 4 ] decomposes N experts into K basis matrices, yielding \mathcal{O}(K\cdot d^{2}+N\cdot K) memory which is a 24–30% reduction. However, this remains linear in N when K scales with expert count. Low rank factorization approaches [ 18 ] achieve 2 \times compression but cannot scale beyond 128 experts on edge constrained devices. • Expert Pruning and merging:PuzzleMoE [ 19 ] merges redundant experts and applies 3-bit quantization, achieving 50% sparsity and a 2 \times with a tradeoff in accuracy. Expert trimming [ 11 ] drops underutilized experts retaining 77% memory but trading efficiency. These approaches share a common structural limitation: expert identity is considered irreducible, which necessitates the explicit materialisation of each weight matrix. 2.2 Quantization for Model Compression Low-bit quantization is a primary compression strategy for large language models. QMoE [ 9 ] achieves sub-1 bit compression through aggressive quantization yielding a 10-20 \times memory reduction with a ¡2% accuracy loss. MoQE [ 15 ] applies 2-bit weight-only quantization to MoE models, achieving a 5.0 \times compression ratio while surpassing dense baselines in accuracy. Mixture Compressor [ 14 ] employs mixed-precision quantization with an average of 2.54 bits, reducing model size to 76.6% of its original size. However, quantization alone cannot break linear scaling. At 2 bits per weight, a 64-expert MoE still requires \frac{N\cdot d^{2}}{16} —e.g., 52 MB for d=512 . Scaling to 256 experts requires 208 MB.Quantization reduces the constant factor but preserves \mathcal{O}(N\cdot d^{2}) complexity. Another limitation arises from outlier handling, activation distributions in transformers exhibit outliers, 10-100x times larger than the median [ 17 ] . QuaRot [ 2 ] applies a single global Hadamard rotation to suppress outliers before quantization. These methods assume outlier distributions remain stable during training,an assumption violated by channel drift during fine tuning. Moreover, global experts cannot capture per-expert specialization, the average case performance is optimized across all experts. 2.3 Structured Parameterizations Butterfly Matrices [ 7 ] parameterize orthogonal transformations with \mathcal{O}(d\log d) parameters by using block-diagonal Givens rotations, enabling fast \mathcal{O}(d\log d) matrix-vector products. To the best of our knowledge, butterfly-structured matrices have not previously been used to parameterize expert identity in MoE architectures. Prior work uses them as drop-in substitutes for dense layers but does not tackle the problem of expert parameterization. LoRA [ 13 ] adds low-rank updates \Delta W_{i} to a frozen backbone, achieving \mathcal{O}(d^{2}+N\cdot d\cdot r) complexity, however this is additive adaptation, and not a fundamental reparameterization of expert capacity. Figure 1 : The proposed Butterfly MoE layer replaces the standard FFN within a Transformer block The missing primitive: a parameterization that represents N experts as group-theoretic variations of a shared substrate, breaking the assumption that experts must be independently stored. 2.4 ButterflyMoE ButterflyMoE introduces a structural insight that experts in MoE models can be parameterized as orbital variations of a single quantized prototype under learned transformations. ButterflyMoE replaces expert storage with a group-orbit parameterization. This eliminates the need to materialize expert weights entirely. Recent work on on linear mode connectivity [ 10 , 1 ] shows that distinct neural networks trained on similar data reside on permutation invariant subspaces. This observation is extended to MoE architecture by parameterizing N independent matrices \{W_{i}\} as: W_{i}=\mathcal{B}(\phi_{i})\cdot W_{\text{base}}\cdot\mathcal{B}(\theta_{i})^{T} where W_{\text{base}}\in\{-1,0,+1\}^{d\times d} is ternary-quantized (1.58 bits/weight) and \mathcal{B}(\phi_{i}),\mathcal{B}(\theta_{i}) are expert-specific Butterfly matrices with \mathcal{O}(\log d) parameters each. This achieves three properties simultaneously: • Sub-linear memory scaling: \mathcal{O}(d^{2}+N\cdot d\log d) versus \mathcal{O}(N\cdot d^{2}) • Dynamic outlier suppression: Learned per-expert input

Read the complete text