Improving Routing in Sparse Mixture of Experts with Graph of Tokens
Tam Nguyen, Ngoc N. Tran, Khai Nguyen, Richard G. Baraniuk · 2025
Read Improving Routing in Sparse Mixture of Experts with Graph of Tokens by Tam Nguyen, Ngoc N. Tran, Khai Nguyen, Richard G. Baraniuk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00792.txt.
Description
Tam Nguyen and co-authors steady the routing inside giant AI models in a 2025 paper. Sparse expert models switch on only a few subnetworks per input, gaining huge size at fixed cost. Late in training, though, the choice of which expert gets a token wobbles, hurting reliability.
Viewing routing through a probability lens, the team blames the fact that tokens are routed independently. Their fix lets similar tokens influence each other's routing, and a second version borrows the attention pattern to guide choices. Theory shows the changes calm the routing distribution, and tests confirm steadier, more accurate, more robust models.
Catalog record
| CREATOR | Tam Nguyen, Ngoc N. Tran, Khai Nguyen, Richard G. Baraniuk |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-00792