Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng · 2025
Read Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts by Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05447.txt.
Description
Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, and Yu Cheng introduce Linear-MoE, a system that joins two fast design trends for large language models, in work from 2025. Linear sequence models handle long inputs at low cost. Mixture of experts layers switch on only a few parts per input, which also saves cost.
Their system has two parts. A modeling part gives one frame that supports all forms of linear sequence model. A training part adds parallel methods, with a sequence parallel scheme built for these models. The team also tests hybrid models that mix their layers with standard transformer expert layers.
Catalog record
| CREATOR | Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-05447