M 2 RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling
Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, Tri Dao · 2026
Read M 2 RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling by Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, Tri Dao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14360.txt.
Description
Mishra and co-authors introduce M2RNN, a recurrent network with matrix-valued states for language modeling. Transformers run in parallel but sit in a low complexity class, which bars tasks like state tracking and code execution. Non-linear RNNs do more, yet their power is capped by state size. M2RNN grows that state as a matrix and uses tensor cores well. It tracks states perfectly at lengths never seen in training. In hybrid models, it beats Gated DeltaNet rivals on perplexity with a third of the state size.
Catalog record
| CREATOR | Mayank Mishra, Shawn Tan, Ion Stoica, Joseph Gonzalez, Tri Dao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77062 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.14360