MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang 1 , Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling 2 , Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu 3 , Blaise Agüera y Arcas, João Sacramento · 2025
Read MesaNet: Sequence Modeling by Locally Optimal Test-Time Training by Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang 1 , Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling 2 , Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu 3 , Blaise Agüera y Arcas, João Sacramento (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05233.txt.
Introduction
Johannes von Oswald and a large team of co-authors introduce a sequence model in twenty twenty-five. Sequence modeling is now ruled by transformers, which use self-attention. At inference, transformers need memory and compute that grow with sequence length. Recent work has linearized that attention step, yielding recurrent models with fixed costs.
Those models, such as DeltaNet and Mamba, can be seen in a shared way. Each derives its update from an in-context regression goal, solved roughly by an online learning rule. The new model, MesaNet, joins that line of work. It is built around locally optimal training at test time. The authors give a version that is numerically stable and can run in parallel chunks.
Catalog record
| CREATOR | Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang 1 , Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling 2 , Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu 3 , Blaise Agüera y Arcas, João Sacramento |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 144 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05233