DOT-MoE: Differentiable Optimal Transport for MoEfication

Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta · 2026

Read DOT-MoE: Differentiable Optimal Transport for MoEfication by Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01666.txt.

Description

Bamba and co-authors convert dense models into sparse experts via optimal transport. MoE designs decouple model size from inference cost. Training them from scratch is unstable and compute-heavy, though. Converting a pre-trained dense model is an attractive alternative. Past conversion uses heuristic neuron clustering or random splitting. Those partition the feed-forward network into experts crudely. DOT-MoE formulates the split as a differentiable optimal transport problem. Neuron assignment becomes a balanced transport task in that view.

Sinkhorn-Knopp iterations enforce strict expert capacity limits. Straight-through estimators learn assignments and routing jointly, end to end. Tests span multiple architectures and benchmarks. DOT-MoE beats pruning, clustering, and random-split baselines clearly. It retains 90 percent of dense performance while halving active parameters.

Catalog record

CREATORUdbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01666

Browse the full library · About the Tower of Babel