Improving Routing in Sparse Mixture of Experts with Graph of Tokens

Tam Nguyen, Ngoc N. Tran, Khai Nguyen, Richard G. Baraniuk · 2025

Read Improving Routing in Sparse Mixture of Experts with Graph of Tokens by Tam Nguyen, Ngoc N. Tran, Khai Nguyen, Richard G. Baraniuk (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00792.txt.

Description

Tam Nguyen and co-authors steady the routing inside giant AI models in a 2025 paper. Sparse expert models switch on only a few subnetworks per input, gaining huge size at fixed cost. Late in training, though, the choice of which expert gets a token wobbles, hurting reliability.

Viewing routing through a probability lens, the team blames the fact that tokens are routed independently. Their fix lets similar tokens influence each other's routing, and a second version borrows the attention pattern to guide choices. Theory shows the changes calm the routing distribution, and tests confirm steadier, more accurate, more robust models.

Catalog record

CREATORTam Nguyen, Ngoc N. Tran, Khai Nguyen, Richard G. Baraniuk
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00792

Browse the full library · About the Tower of Babel