Emo : Pretraining Mixture of Experts for Emergent Modularity

Ryan Wang Akshita Bhagia Sewon Min, University of California, Berkeley, Allen Institute for AI, [email protected]@[email protected] · 2026

Read Emo : Pretraining Mixture of Experts for Emergent Modularity by Ryan Wang Akshita Bhagia Sewon Min, University of California, Berkeley, Allen Institute for AI, [email protected]@[email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06663.txt.

Description

Ryan Wang, Akshita Bhagia, and Sewon Min introduce EMO, a mixture-of-experts model designed so parts of it can be used independently. Large models are usually deployed whole, even when an application needs only code or math ability. Standard expert models degrade badly when restricted to a subset of experts, wasting their apparent modularity, especially on memory-limited hardware.

EMO encourages tokens from similar domains to share expert pools by restricting each document to its own pool during pretraining, using document boundaries alone. A model with one billion active parameters matches standard performance, and keeping only a quarter of its experts costs about one percent accuracy. Its experts specialize by meaning, such as domain, not just syntax.

Catalog record

CREATORRyan Wang Akshita Bhagia Sewon Min, University of California, Berkeley, Allen Institute for AI, [email protected]@[email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06663

Browse the full library · About the Tower of Babel