Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts

Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng · 2025

Read Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts by Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05447.txt.

Description

Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, and Yu Cheng introduce Linear-MoE, a system that joins two fast design trends for large language models, in work from 2025. Linear sequence models handle long inputs at low cost. Mixture of experts layers switch on only a few parts per input, which also saves cost.

Their system has two parts. A modeling part gives one frame that supports all forms of linear sequence model. A training part adds parallel methods, with a sequence parallel scheme built for these models. The team also tests hybrid models that mix their layers with standard transformer expert layers.

Catalog record

CREATORWeigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05447

Browse the full library · About the Tower of Babel