It’s All Connected : A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni · 2025

Read It’s All Connected : A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization by Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13173.txt.

Introduction

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, and Vahab Mirrokni rethink how sequence models store and recall facts. Their two thousand twenty-five paper draws on attentional bias, the human habit of favoring some events over others. They view transformers and newer recurrent networks as memory modules. Each one learns a map from keys to values under an inner goal.

The authors find that most models use one of just two inner goals. One is dot-product similarity, and the other is a squared-error fit. They propose fresh choices for this inner goal, with stable ways to train them. They also recast forgetting as part of the same frame. The paper links test-time memory, retention, and online learning into one shared picture.

Catalog record

CREATORAli Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13173

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

It’s All Connected : A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni Abstract Designing efficient and effective architectural backbones has been in the core of research efforts to enhance the capability of foundation models. Inspired by the human cognitive phenomenon of attentional bias—the natural tendency to prioritize certain events or stimuli—we reconceptualize neural architectures, including Transformers, Titans, and modern linear recurrent neural networks as associative memory modules that learn a mapping of keys and values using an internal objective, referred to as attentional bias . Surprisingly, we observed that most existing sequence models leverage either (1) dot-product similarity, or (2) ℓ 2 \ell_{2} regression objectives as their attentional bias. Going beyond these objectives, we present a set of alternative attentional bias configurations along with their effective approximations to stabilize their training procedure. We then reinterpret forgetting mechanisms in modern deep learning architectures as a form of retention regularization, providing a novel set of forget gates for sequence models. Building upon these insights, we present Miras , a general framework to design deep learning architectures based on four choices of: (i) associative memory architecture, (ii) attentional bias objective, (iii) retention gate, and (iv) memory learning algorithm. We present three novel sequence models— Moneta , Yaad , and Memora —that go beyond the power of existing linear RNNs while maintaining a fast parallelizable training process. Our experiments show different design choices in Miras yield models with varying strengths. For example, certain instances of Miras achieve exceptional performance in special tasks such as language modeling, commonsense reasoning, and recall intensive tasks, even outperforming Transformers and other modern linear recurrent models. 1 Introduction Designing efficient architectural backbones for sequence modeling is a key to enhance the capability of foundation models in domains ranging from language ( Vaswani et al., 2017 ; Behrouz et al., 2024b ) and computer vision ( Dosovitskiy et al., 2020 ) to computational biology ( Wang et al., 2024 ) and neuroscience ( Behrouz et al., 2024 ) . While Transformers ( Vaswani et al., 2017 ) , mainly due to their in-context learning and ability to learn at scale ( Kaplan et al., 2020 ) , have been firmly established as state-of-the-art (SOTA) models in sequence modeling, their quadratic time and space complexity limits their applicability in tasks that require long context modeling ( Dalal et al., 2025 ; Liu et al., 2024a ; Li et al., 2024 ) . Recent efforts aim to overcome Transformer limitations in long-context modeling by designing efficient recurrent alternatives ( Neil et al., 2017 ; Smith et al., 2022 ; Behrouz et al., 2024b ) . Unlike Transformer’s linearly growing memory (i.e., the KV cache), these models compress the context into a fixed size memory, demanding improved memory management for comparable performance. To design more effective architectures, studies focus on improving memory capacity and its management by using/designing more expressive: (1) Learning rules: from Hebbian rule ( Hebb, 2005 ) to Delta rule ( Neil et al., 2017 ) ; (2) Forget gates: from LSTM’s ( Schmidhuber & Hochreiter, 1997 ) to Mamba2’s ( Dao & Gu, 2024 ) and then Titan’s forget gates ( Behrouz et al., 2024b ) ; and (3) More expressive memory architectures: from vector-valued memory in RetNet ( Sun et al., 2023 ) and LRU ( Orvieto et al., 2023 ) to neural deep memory in Titans ( Behrouz et al., 2024b ) and TTT ( Sun et al., 2024 ) . At the core of these advancements lies a critical question: “what is the underlying design framework behind these sequence models, and how can these models be enhanced?”. Taking inspiration from the broad definitions of associative memory and learning in neuropsychology literature ( Okano et al., 2000 ) , several studies discuss the connection between Transformers and (linear) Recurrent Neural Networks (RNNs) with associative memory ( Hopfield, 1982 ; Ramsauer et al., 2021 ; Bietti et al., 2023 ) . These studies, however, either: (1) lack a universal explanation to fully illustrate the underlying learning algorithms, (2) are limited to a specific definition of associative memory and lack generalizability, and/or (3) are unable to describe standard, widely used components such as forget gate. Contributions. Inspired by the human cognitive phenomenon of attentional bias—the natural tendency to prioritize certain events or stimuli—we re-conceptualize neural architectures, including Transformers, Titans, and other modern linear recurrent neural networks based on a broad definition of associative memory with attentional bias . We define and formalize the concept of attentional bias as the internal memory objective of sequence models (see Section 3 ) that aims to learn the underlying mapping between inputs (i.e., keys and values). Our formulation reveals that almost all existing sequence models are associative memories that leverage the same type of attentional bias. We reinterpret existing forgetting mechanisms in modern deep learning architectures as a form of retention \ell_{2} -regularization for the attentional bias, and then provide a novel set of alternative retention gates (forget gate) for sequence models, providing new insights on how to balance learning new concepts and the retention of previously learned concepts. Building upon our formulation of memory and forget gate, we present Miras 1 1 1 “Miras” is the translation of “Legacy” in several languages: such as Persian, Arabic, and Turkish. We choose this name since this framework provides clear steps for future studies to design powerful sequence models based on their task at hand. , a fundamental framework to design novel sequence modeling architectures by four choice of: (1) Attentional bias (i.e., memory objective), (2) Retention gate, (3) Memory architecture, and (4) Memory learning algorithm (i.e., optimizer). We motivate and discuss several novel design choices, leading to novel architectures beyond existing sequence modeling architectures. Finally, we focus on three novel variants of Miras — Moneta , Yaad , and Memora —that are based on attentional biases beyond simple \ell_{2} -regression objective as well as novel retention gating mechanisms that are more robust than existing ones. We further perform experimental evaluations of these three variants on language modeling, common-sense reasoning, needle-in-haystack, and recall intensive tasks. The results illustrates the superior performance of these variants, outperforming state-of-the-art sequence models. Roadmap. In Section 2 , we review literature and discuss relevant concepts that we use through the paper. In Section 3 , we present and discuss the broad definition of associative memory with formally defining the concept of attentional bias . We then discuss two viewpoints—Learning-Retaining and Follow-the-Regularized-Leader (FTRL)—to interpret sequence modeling through the lens of optimization and prove the generality of Learning-Retaining over FTRL. In Section 4 , we present our Miras framework and discuss how it unifies modern sequence models. In Section 5 , to show the potential of Miras framework, we discuss a variety of novel design choices for (1) attentional bias, and (2) retention gate (forget gate). Later in Section 5.3 , we present three novel sequence models as the variants of Miras , and then discuss how to train them in a parallelizable manner. Finally, our experimental evaluations are reported in Section 6 . Figure 1 : The overview of Miras framework. Miras is based on four critical choices of (1)

Read the complete text