Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner

Liu Xiao, Li Zhiyuan, Lin Yueyu · 2025

Read Millions of States: Designing a Scalable MoE Architecture with RWKV-7 Meta-learner by Liu Xiao, Li Zhiyuan, Lin Yueyu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08247.txt.

Description

Liu Xiao, Li Zhiyuan, and Lin Yueyu grow a state-based sequence model so it can scale like its transformer kin. State models run in a straight line with fixed memory, a cheap trick, but they lack a way for tokens and weights to talk. Their 2025 design, Meta-State, builds that talk into the state itself.

Part of the model's state doubles as change weights, with no new matrices and no softmax step. Token order stays strict, as live text needs. Growth then means widening the state and the weight tokens, reusing all old weights. No retraining starts the scale-up. The team frames it as a bridge: state speed, weight talk, and room to grow in one frame.

Catalog record

CREATORLiu Xiao, Li Zhiyuan, Lin Yueyu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08247

Browse the full library · About the Tower of Babel