Hyperparameter Transfer with Mixture-of-Experts Layers
Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin · 2026
Read Hyperparameter Transfer with Mixture-of-Experts Layers by Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20205.txt.
Description
Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, and Boris Hanin study how to carry tuning settings across model scales in mixture-of-experts networks. These networks route each token to a few expert subnetworks, which separates total size from the compute used per token. The design adds new knobs, such as router weights and the number and size of experts.
Each knob needs tuning, and tuning at full scale is costly. The authors examine how good settings found on small models transfer to large ones with expert layers. Reliable transfer rules would cut the cost of training the sparse models now common in large AI systems.
Catalog record
| CREATOR | Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-20205