M 2 RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling

Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, Tri Dao · 2026

Read M 2 RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling by Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, Tri Dao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14360.txt.

Description

Mishra and co-authors introduce M2RNN, a recurrent network with matrix-valued states for language modeling. Transformers run in parallel but sit in a low complexity class, which bars tasks like state tracking and code execution. Non-linear RNNs do more, yet their power is capped by state size. M2RNN grows that state as a matrix and uses tensor cores well. It tracks states perfectly at lengths never seen in training. In hybrid models, it beats Gated DeltaNet rivals on perplexity with a third of the state size.

Catalog record

CREATORMayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, Tri Dao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77062
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14360

Browse the full library · About the Tower of Babel