Hydra: A 1.6B-Parameter State-Space Language Model with Sparse Attention, Mixture-of-Experts, and Memory
Siddharth Chaudhary [email protected], Bennett Browning [email protected] · 2025
Read Hydra: A 1.6B-Parameter State-Space Language Model with Sparse Attention, Mixture-of-Experts, and Memory by Siddharth Chaudhary [email protected], Bennett Browning [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15099.txt.
Description
Siddharth Chaudhary and co-authors published a 2025 paper presenting Hydra, a language model design built for long documents on modest hardware. The standard transformer design grows costly very fast as text lengthens, which blocks use in tight settings. Hydra swaps the core for a state-space backbone, a design whose cost grows gently with length. Extra parts are added modularly.
The model routes work between sparse attention for long-range links, a mixture of expert sub-networks, and two kinds of memory: a scratch workspace for reasoning and a key-based store for lookup. The team evaluated a 29-million-parameter version. It processed text about three times faster than equal-sized transformers at eight thousand tokens. On multi-step logic chains, its accuracy rose tenfold. Removing each part in turn confirmed each one earns its place.
Catalog record
| CREATOR | Siddharth Chaudhary [email protected], Bennett Browning [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15099