DOT-MoE: Differentiable Optimal Transport for MoEfication
Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta · 2026
Read DOT-MoE: Differentiable Optimal Transport for MoEfication by Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01666.txt.
Description
Bamba and co-authors convert dense models into sparse experts via optimal transport. MoE designs decouple model size from inference cost. Training them from scratch is unstable and compute-heavy, though. Converting a pre-trained dense model is an attractive alternative. Past conversion uses heuristic neuron clustering or random splitting. Those partition the feed-forward network into experts crudely. DOT-MoE formulates the split as a differentiable optimal transport problem. Neuron assignment becomes a balanced transport task in that view.
Sinkhorn-Knopp iterations enforce strict expert capacity limits. Straight-through estimators learn assignments and routing jointly, end to end. Tests span multiple architectures and benchmarks. DOT-MoE beats pruning, clustering, and random-split baselines clearly. It retains 90 percent of dense performance while halving active parameters.
Catalog record
| CREATOR | Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01666