Attention to Mamba: A Recipe for Cross-Architecture Distillation
Abhinav Moudgil †∗, Ningyuan Huang ‡∗, Eeshan Gunesh Dhekane, Pau Rodríguez, Luca Zappella, Federico Danieli · 2026
Read Attention to Mamba: A Recipe for Cross-Architecture Distillation by Abhinav Moudgil †∗, Ningyuan Huang ‡∗, Eeshan Gunesh Dhekane, Pau Rodríguez, Luca Zappella, Federico Danieli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14191.txt.
Description
Moudgil and five colleagues distill an attention-based Transformer into a Mamba-style state space model without any attention blocks. A naive transfer fails to preserve teacher quality, so they stage the transfer in two steps. First they distill the Transformer into a linearized attention form with a kernel-trick adaptation, then distill that form into an adapted Mamba. The distilled model keeps the Pythia-1B teacher's downstream quality, and ablations at 1B scale with 10B tokens test mixer designs, scaling, and token splits.
Catalog record
| CREATOR | Abhinav Moudgil †∗, Ningyuan Huang ‡∗, Eeshan Gunesh Dhekane, Pau Rodríguez, Luca Zappella, Federico Danieli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57965 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14191