Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh, Ziteng Sun · 2026
Read Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations by Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh, Ziteng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28769.txt.
Description
Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh, and Ziteng Sun propose Oryx, a hybrid model switching between mixers across a token sequence. Softmax attention costs linear memory and quadratic compute, while linear recurrent models lag on long-context retrieval and in-context tasks. Oryx ties at least 90 percent of parameters across mixers, letting attention and recurrent modes share representations. With Mamba-2 and Gated DeltaNet variants up to 1.4B, Oryx matches or beats single-mixer baselines under fixed token budgets, and at 1.4B every instance beats its counterpart.
Catalog record
| CREATOR | Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh, Ziteng Sun |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59367 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28769