Hydra: A 1.6B-Parameter State-Space Language Model with Sparse Attention, Mixture-of-Experts, and Memory

Siddharth Chaudhary [email protected], Bennett Browning [email protected] · 2025

Read Hydra: A 1.6B-Parameter State-Space Language Model with Sparse Attention, Mixture-of-Experts, and Memory by Siddharth Chaudhary [email protected], Bennett Browning [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15099.txt.

Description

Siddharth Chaudhary and co-authors published a 2025 paper presenting Hydra, a language model design built for long documents on modest hardware. The standard transformer design grows costly very fast as text lengthens, which blocks use in tight settings. Hydra swaps the core for a state-space backbone, a design whose cost grows gently with length. Extra parts are added modularly.

The model routes work between sparse attention for long-range links, a mixture of expert sub-networks, and two kinds of memory: a scratch workspace for reasoning and a key-based store for lookup. The team evaluated a 29-million-parameter version. It processed text about three times faster than equal-sized transformers at eight thousand tokens. On multi-step logic chains, its accuracy rose tenfold. Removing each part in turn confirmed each one earns its place.

Catalog record

CREATORSiddharth Chaudhary [email protected], Bennett Browning [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15099

Browse the full library · About the Tower of Babel