∞ \infty -MoE: Generalizing Mixture of Experts to Infinite Experts

Shota Takashiro , Takeshi Kojima , Shohei Taniguchi , Yusuke Iwasawa , Yutaka Matsuo · 2026

Read ∞ \infty -MoE: Generalizing Mixture of Experts to Infinite Experts by Shota Takashiro , Takeshi Kojima , Shohei Taniguchi , Yusuke Iwasawa , Yutaka Matsuo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17680.txt.

Description

Shota Takashiro and four co-authors generalize Mixture of Experts to infinitely many experts. Standard MoE picks a few feed-forward networks per token. That gives a good trade of cost and quality. In standard designs, each expert stands alone. Experts combine in a discrete space. As expert counts grow, training each one well gets hard.

Their infinite form rethinks that structure. A continuous view of experts could scale more smoothly. Language models lean on MoE for cheap capacity growth. A better expert frame could cut training waste. The work asks what experts should be at the limit. It may reshape how large sparse models are built.

Catalog record

CREATORShota Takashiro , Takeshi Kojima , Shohei Taniguchi , Yusuke Iwasawa , Yutaka Matsuo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17680

Browse the full library · About the Tower of Babel