∞ \infty -MoE: Generalizing Mixture of Experts to Infinite Experts
Shota Takashiro , Takeshi Kojima , Shohei Taniguchi , Yusuke Iwasawa , Yutaka Matsuo · 2026
Read ∞ \infty -MoE: Generalizing Mixture of Experts to Infinite Experts by Shota Takashiro , Takeshi Kojima , Shohei Taniguchi , Yusuke Iwasawa , Yutaka Matsuo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17680.txt.
Description
Shota Takashiro and four co-authors generalize Mixture of Experts to infinitely many experts. Standard MoE picks a few feed-forward networks per token. That gives a good trade of cost and quality. In standard designs, each expert stands alone. Experts combine in a discrete space. As expert counts grow, training each one well gets hard.
Their infinite form rethinks that structure. A continuous view of experts could scale more smoothly. Language models lean on MoE for cheap capacity growth. A better expert frame could cut training waste. The work asks what experts should be at the limit. It may reshape how large sparse models are built.
Catalog record
| CREATOR | Shota Takashiro , Takeshi Kojima , Shohei Taniguchi , Yusuke Iwasawa , Yutaka Matsuo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-17680