DYNAMAX: Dynamic computing for Transformers and Mamba based architectures

Miguel Nogales 1 , Matteo Gambella 2 , and Manuel Roveri 2 · 2025

Read DYNAMAX: Dynamic computing for Transformers and Mamba based architectures by Miguel Nogales 1 , Matteo Gambella 2 , and Manuel Roveri 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20922.txt.

Description

Miguel Nogales, Matteo Gambella, and Manuel Roveri let language models answer early when they are already confident, and recruit a rival architecture as the judge, in a 2025 paper. Early exits cut compute and delay, but most work covers encoder-only models. Decoder models and the newer Mamba family were largely unexplored.

Their framework, DYNAMAX, builds exits into Mamba models by exploiting their dynamic design. More novel, it also uses a Mamba model as the exit classifier for both Mamba and transformer systems. Tests pitted a Mistral transformer against a Mamba-based model on question answering datasets. The Mamba judge adapted well, holding accuracy while saving computation. The authors aim at embedded and resource-limited settings where every spared cycle counts. They present Mamba as a general tool for dynamic computing, not just a model family.

Catalog record

CREATORMiguel Nogales 1 , Matteo Gambella 2 , and Manuel Roveri 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20922

Browse the full library · About the Tower of Babel