Characterizing Mamba’s Selective Memory using Auto-Encoders

Tamanna Hossain Robert L. Logan IV Ganesh Jagadeesan, Sameer Singh Joel Tetreault Alejandro Jaimes · 2025

Read Characterizing Mamba’s Selective Memory using Auto-Encoders by Tamanna Hossain Robert L. Logan IV Ganesh Jagadeesan, Sameer Singh Joel Tetreault Alejandro Jaimes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15653.txt.

Introduction

Tamanna Hossain and co authors probe what the Mamba language model family forgets in a 2025 paper. Mamba models use a fixed size memory at run time, unlike transformers, so long inputs must lose some information. Earlier work timed when loss begins, but not which kinds of information go first.

The team trains an autoencoder to rebuild sequences from Mamba's hidden state and compares inputs with rebuilds for models from 130 million to 1.4 billion parameters. Loss falls hardest on math tokens such as numbers and variables, on organization names, and on dialects other than Standard American English. Forgotten tokens tend to be rarer in pretraining data, which points to clear fixes.

Catalog record

CREATORTamanna Hossain Robert L. Logan IV Ganesh Jagadeesan, Sameer Singh Joel Tetreault Alejandro Jaimes
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-15653

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Characterizing Mamba’s Selective Memory using Auto-Encoders Tamanna Hossain Robert L. Logan IV Ganesh Jagadeesan, Sameer Singh Joel Tetreault Alejandro Jaimes Abstract State space models (SSMs) are a promising alternative to transformers for language modeling because they use fixed memory during inference. However, this fixed memory usage requires some information loss in the hidden state when processing long sequences. While prior work has studied the sequence length at which this information loss occurs, it does not characterize the types of information SSM language models (LMs) tend to forget. In this paper, we address this knowledge gap by identifying the types of tokens (e.g., parts of speech, named entities) and sequences (e.g., code, math problems) that are more frequently forgotten by SSM LMs. We achieve this by training an auto-encoder to reconstruct sequences from the SSM’s hidden state, and measure information loss by comparing inputs with their reconstructions. We perform experiments using the Mamba family of SSM LMs (130M--1.4B) on sequences ranging from 4--256 tokens. Our results show significantly higher rates of information loss on math-related tokens (e.g., numbers, variables), mentions of organization entities, and alternative dialects to Standard American English. We then examine the frequency that these tokens appear in Mamba’s pretraining data and find that less prevalent tokens tend to be the ones Mamba is most likely to forget. By identifying these patterns, our work provides clear direction for future research to develop methods that better control Mamba’s ability to retain important information. 1 1 1 Our code is available at: https://github.com/dataminr-ai/ouroboros . 1 Introduction Figure 1: Identifying Selective Memory Loss. We train auto-encoders to reconstruct inputs directly from Mamba’s hidden states, and measure information loss by comparing inputs to their reconstructions. These reconstructions act as probes of the hidden state’s information retention: more faithful reconstruction implies greater information retention. State space models (SSMs) have emerged as a promising alternative to transformers ( 31 ) for language modeling, with the Mamba ( 5 ) SSM LMs achieving comparable performance to similar sized transformer models while being substantially more memory efficient at inference time. This memory efficiency arises from the fact that SSM next token predictions are computed recurrently using a fixed-size hidden state, whereas transformer next token predictions are computed using a key-value cache that grows linearly with the sequence length. Recent work has shown that this efficiency comes at the cost of some information loss when processing long sequences. In particular, 12 establish an upper bound on the length of sequences that can be exactly copied by Mamba, while 35 prove that the influence of an input token decays exponentially with input length. To provide empirical support for these theoretical results, these works measure Mamba’s decline in accuracy as sequence length increases on copying tasks that either use synthetic inputs, or prime the model to memorize information using cues in the prompt. While these results characterize when Mamba begins to forget, they leave open the question of what kinds of information is more likely to be forgotten, particularly on natural inputs. In this paper, we develop a methodology that allows us to better understand the types of tokens and sequences that the Mamba LM is likely to forget in practical settings. The core idea of our approach is to compare input texts with known properties—such as named entity recognition (NER) and parts of speech (POS) tags on the token level, and categorical labels on the sequence level—to reconstructions obtained from Mamba’s hidden state. By correlating reconstruction errors (measured using F1 scores and token omission rates) with the input properties we are able to obtain high-level insights about Mamba’s selective memory . For example, in Figure 1 we see that the number "42" is reconstructed as "57", providing some evidence that Mamba has a tendency to forget numbers. To obtain input reconstructions, we train an auto-encoder that decodes sequences from hidden states produced by a frozen Mamba encoder. By using pretrained checkpoints for our encoder, we are able to examine the impact language model pretraining on Mamba’s memory, rather than just the architecture. This approach additionally complements the approaches used by prior works in that it allows us to study reconstructions of any piece of text without having to provide memorization cues. In Section 4.2 we measure the performance of this approach for a number of different model sizes and sequence lengths, and validate that it replicates findings from previous works. In Sections 4.3 and 4.4 we then employ this approach to study token- and sequence-level characteristics of reconstruction errors on a number of datasets. We begin by looking at Mamba’s pretraining dataset The Pile ( 3 ) , which allows us to understand how in-distribution errors vary by the input source. Our results show that Mamba is significantly more likely to forget tokens on math-related sequences, even for relatively short sequence lengths (16+ tokens). We also perform experiments on 4 ’s SAE/AAVE tweet pair dataset, and find significantly elevated reconstruction errors on texts written using African American Vernacular English. We then study token-level NER and POS errors on CoNLL-2003 ( 28 ) , and find significantly higher reconstruction errors on mentions of organizations, as well as numerical tokens. To better understand this, we examine a 178M token sample from the Pile and find an association between the categories that Mamba tends to forget and the frequency that unique tokens belonging to those categories appear in the Pile, suggesting that their rarity in Mamba’s pre-training data may contribute to their omission. This raises the question of whether conventional pretraining pipelines are well suited for the inductive biases of SSM-based language models. In sum, our work establishes clear differences in the types of information the Mamba LM tends to retain versus forget, providing direction for future research into methods to improve the retention of important information in its hidden state. 2 Related Work State Space Models State space models (SSMs) are linear recurrent architectures with fixed-size hidden states that enable linear-time training and constant-time inference per step, making them a more efficient alternative to transformers for sequence modeling ( 7 ; 6 ) . Recent models like Mamba ( 5 ) have shown that SSMs can match transformers of similar scale on standard NLP benchmarks, aided by long-range initialization techniques from HiPPO theory ( 8 ) and time-dependent parameterization that allows selective attention over inputs. Hybrid transformer-SSM architectures have shown gains in both efficiency and accuracy over pure transformers ( 34 ) , reinforcing the value of SSM-based models—especially as inference-time scaling is increasingly used to enhance LM reasoning through techniques like task decomposition ( 37 ; 39 ) and increased sampling ( 36 ) . Although Mamba models offer improved computational efficiency over transformers, recent studies have shown that this comes at the cost of information degradation over long contexts. 12 derive a theoretic upper bound on the sequence length that Mamba can recall information from, and 35 demonstrate that token influence decays exponentially with input distance, i.e., there is a recency bias to Mamba’s memory. These findings are empirically supported by experiments using input sequences with explicit memorization cues and synthetic inputs ( 32 ; 35 ) . However, while these prior works illuminate when forgetting occurs in Mamba’s

Read the complete text