Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner
Liu Xiao, Li Zhiyuan, Lin Yueyu · 2025
Read Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner by Liu Xiao, Li Zhiyuan, Lin Yueyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08247.txt.
Description
Liu Xiao, Li Zhiyuan, and Lin Yueyu grow a state-based sequence model so it can scale like its transformer kin. State models run in a straight line with fixed memory, a cheap trick, but they lack a way for tokens and weights to talk. Their 2025 design, Meta-State, builds that talk into the state itself.
Part of the model's state doubles as change weights, with no new matrices and no softmax step. Token order stays strict, as live text needs. Growth then means widening the state and the weight tokens, reusing all old weights. No retraining starts the scale-up. The team frames it as a bridge: state speed, weight talk, and room to grow in one frame.
Catalog record
| CREATOR | Liu Xiao, Li Zhiyuan, Lin Yueyu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08247