DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
Gökdeniz Gülmez · 2026
Read DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks by Gökdeniz Gülmez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01697.txt.
Description
Gokdeniz Gulmez loosens two rigid habits in mixture-of-experts networks in this 2026 work. Such networks split work among expert sub-models, yet most designs wake a fixed count of experts per token. They also spread expert capacity evenly across all layers, as if each layer needed the same help.
His frame, DynaMoE, lets the active expert count vary with how hard each token is. Capacity per layer follows one of six schedules, such as descending, ascending, pyramid, or wave shapes. Theory work backs gains in expressive power and gives bounds on compute cost. Tests span image tasks and language modeling at several model sizes. The best schedule depends on task and scale, with early-heavy layouts winning on image tasks in the tests.
Catalog record
| CREATOR | Gökdeniz Gülmez |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72416 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.01697