DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

Gökdeniz Gülmez · 2026

Read DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks by Gökdeniz Gülmez (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01697.txt.

Description

Gokdeniz Gulmez loosens two rigid habits in mixture-of-experts networks in this 2026 work. Such networks split work among expert sub-models, yet most designs wake a fixed count of experts per token. They also spread expert capacity evenly across all layers, as if each layer needed the same help.

His frame, DynaMoE, lets the active expert count vary with how hard each token is. Capacity per layer follows one of six schedules, such as descending, ascending, pyramid, or wave shapes. Theory work backs gains in expressive power and gives bounds on compute cost. Tests span image tasks and language modeling at several model sizes. The best schedule depends on task and scale, with early-heavy layouts winning on image tasks in the tests.

Catalog record

CREATORGökdeniz Gülmez
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72416
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01697

Browse the full library · About the Tower of Babel