Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations

Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh, Ziteng Sun · 2026

Read Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations by Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh, Ziteng Sun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28769.txt.

Description

Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh, and Ziteng Sun propose Oryx, a hybrid model switching between mixers across a token sequence. Softmax attention costs linear memory and quadratic compute, while linear recurrent models lag on long-context retrieval and in-context tasks. Oryx ties at least 90 percent of parameters across mixers, letting attention and recurrent modes share representations. With Mamba-2 and Gated DeltaNet variants up to 1.4B, Oryx matches or beats single-mixer baselines under fixed token budgets, and at 1.4B every instance beats its counterpart.

Catalog record

CREATORKevin Y. Li, Asher Trockman, Ananda Theertha Suresh, Ziteng Sun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59367
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28769

Browse the full library · About the Tower of Babel