Titans: Learning to Memorize at Test Time

Ali Behrouz, Peilin Zhong, Vahab Mirrokni · 2025

Read Titans: Learning to Memorize at Test Time by Ali Behrouz, Peilin Zhong, Vahab Mirrokni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00663.txt.

Description

Ali Behrouz, Peilin Zhong, and Vahab Mirrokni give neural networks a long-term memory that keeps learning while the model is already running. Attention reads the whole context exactly but pays a quadratic price, while recurrent models squeeze the past into a fixed state and lose detail. Titans split the labor between the two.

Their 2024 paper builds a memory module that memorizes history at test time through its own fast update rule, trainable in parallel. Attention then plays short-term memory over the present window, and the learned module holds the long past. Three architecture variants wire the pair together in different ways. Across language, reasoning, genomics, and time series tests, Titans beat Transformers and recent recurrent rivals at long range.

Catalog record

CREATORAli Behrouz, Peilin Zhong, Vahab Mirrokni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-00663

Browse the full library · About the Tower of Babel