Hyperparameter Transfer with Mixture-of-Experts Layers

Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin · 2026

Read Hyperparameter Transfer with Mixture-of-Experts Layers by Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20205.txt.

Description

Tianze Jiang, Blake Bordelon, Cengiz Pehlevan, and Boris Hanin study how to carry tuning settings across model scales in mixture-of-experts networks. These networks route each token to a few expert subnetworks, which separates total size from the compute used per token. The design adds new knobs, such as router weights and the number and size of experts.

Each knob needs tuning, and tuning at full scale is costly. The authors examine how good settings found on small models transfer to large ones with expert layers. Reliable transfer rules would cut the cost of training the sparse models now common in large AI systems.

Catalog record

CREATORTianze Jiang, Blake Bordelon, Cengiz Pehlevan, Boris Hanin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20205

Browse the full library · About the Tower of Babel