Quantifying Memory Utilization with Effective State-Size
Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, Stefano Massaroli · 2025
Read Quantifying Memory Utilization with Effective State-Size by Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, Stefano Massaroli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-19561.txt.
Introduction
Rom Parnichkun and eight co-authors propose a number that measures how much of its memory a sequence model actually uses, in a 2025 paper. Models like attention networks and recurrent networks store past information to shape future output, but designs are hard to compare. Raw attention maps mislead, and cache size only states capacity, not use.
Borrowing from signal processing and control theory, they define effective state-size. The measure fits a broad family of systems, covering attention variants, convolutions, and recurrences. It proves practical: it guides better starting weights, suggests new penalty terms for training, and helps squeeze big models into small ones without losing the performance edge. It also reveals that separator tokens, like end-of-speech markers, affect memory use differently across architectures. The authors offer the metric as a shared yardstick for designing leaner sequence models.
Catalog record
| CREATOR | Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, Stefano Massaroli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 111 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-19561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Quantifying Memory Utilization with Effective State-Size Rom N. Parnichkun, Neehal Tumma, Armin W. Thomas, Alessandro Moro, Qi An, Taiji Suzuki, Atsushi Yamashita, Michael Poli, Stefano Massaroli Abstract The need to develop a general framework for architecture analysis is becoming increasingly important, given the expanding design space of sequence models. To this end, we draw insights from classical signal processing and control theory, to develop a quantitative measure of memory utilization : the internal mechanisms through which a model stores past information to produce future outputs. This metric, which we call effective state-size (ESS), is tailored to the fundamental class of systems with input-invariant and input-varying linear operators , encompassing a variety of computational units such as variants of attention, convolutions, and recurrences. Unlike prior work on memory utilization, which either relies on raw operator visualizations (e.g. attention maps), or simply the total memory capacity (i.e. cache size) of a model, our metrics provide highly interpretable and actionable measurements. In particular, we show how ESS can be leveraged to improve initialization strategies, inform novel regularizers and advance the performance-efficiency frontier through model distillation. Furthermore, we demonstrate that the effect of context delimiters (such as end-of-speech tokens) on ESS highlights cross-architectural differences in how large language models utilize their available memory to recall information. Overall, we find that ESS provides valuable insights into the dynamics that dictate memory utilization, enabling the design of more efficient and effective sequence models. † † affiliationnotice: Equal contribution Equal senior contribution 1 Introduction In recent years, the success of autoregressive sequence modeling in the context of deep learning has largely been driven by advancements in highly parallelizable causal architectures, such as the transformer ( Vaswani et al., 2023 ) . However, despite their strong performance and hardware efficiency, understanding the inner workings of these neural networks remains a challenging task due to their non-linearity and the diversity of fundamental building blocks used. To this end, we leverage a new class of model abstractions, allowing for the development of a unified framework for the analysis of these computational units. In particular, we note that the majority of sequence models of practical interest can formally be expressed as either linear systems ( y=Tu ) or systems with input-varying linear operators ( y=f_{T}(u)u ), the latter of which we abbreviate as LIV 1 1 1 We abbreviate them as LIVs instead of IVLs to conform to the classical convention, where linear time-varying and time-invariant systems are abbreviated as LTVs and LTIs respectively, as well as recent works such as ( Zancato et al., 2024 ) , which call them “linear input-varying systems”. . LIVs generalize the notion of adaptive, or data-controlled operators to a broader class than previously described in ( Massaroli et al., 2021 ; Poli et al., 2023 ) . The input-varying linear operator framework decouples the input-varying featurization u\mapsto T\coloneqq f_{T}(u) and the linear mapping y=Tu required to construct and apply the operator respectively. This decomposition enables a wide array of deep learning primitives to be uniformly formulated as linear systems, including models like convolutions [ 40 ; 47 ; 35 ; 56 ], linear state-transition recurrences [ 15 ; 18 ; 27 ; 61 ; 29 ; 26 ; 75 ; 49 ; 16 ], and attention variants [ 68 ; 34 ; 65 ]. Current approaches to analyzing the inner workings of LIVs often rely upon simple visualizations of the materialized operator T (or the aggregation of T across multiple layers and residuals) [ 45 ; 69 ; 1 ; 4 ; 74 ; 63 ]. However, these visualizations alone often fail to highlight critical properties that explain how different models construct internal representations of the input data. Moreover, prior attempts in obtaining quantitative metrics, such as through spectral analysis of the operator T ( Min & Li, 2024 ; Bhojanapalli et al., 2020 ) , are either limited to a specific model class or do not appropriately take into account important conflating factors like the causal masking of T which significantly distorts the metric ( Wu et al., 2024 ) . In this work, we focus our analysis on the working memory 2 2 2 Here, we refer to “memory” in the sense commonly associated with the “state” of dynamical systems, as described by ( Willems, 1989 ) , as opposed to the notion of language models memorizing some fact encountered during training ( Allen-Zhu & Li, 2024 ) . of model architectures and examine two aspects of model memory in particular: memory capacity (i.e. cache/state size) and memory utilization . Notably, memory capacity alone can be misleading, as models with similar capacities may learn to utilize their available memory to varying degrees. Therefore, we introduce the notion of memory utilization – a measure that provides deeper insight into the differences between architectures with comparable computational efficiency. Our main technical contributions can be summarized as follows: • We draw from classical signal processing and control theory, and propose effective state-size (ESS) – a metric computed by taking the rank of T_{i:,:i-1} – as a proxy for memory utilization in LIVs (Section 3 ). • We validate ESS beyond its theoretical interpretation by demonstrating its correlation with performance across a wide range of models and memory-intensive synthetic tasks, including associative recall and selective copying (Section 4 ). • We explore the use of the ESS metric as a means of enhancing the performance-efficiency trade-off by demonstrating its ability to inform model distillation (Section 5.1 ), initialization schemes (Section A ), and regularization strategies (Section B ). • We extend the utility of ESS to language, demonstrating how it captures a previously uncharacterized property of LLMs: state modulation (Section 5.2 ). This phenomenon provides concrete intuition as to why introducing input dependencies into state transitions is crucial for in-context recall. Figure 1 : An overview of the effective state-size metric and its various downstream applications. 2 Related Work In this section, we briefly describe previous works that are closely related to the core findings of the paper. For an extended discussion of the related works, refer to Section C . Classifying sequence models. A sequence model can be defined as a mapping from input sequences to output sequences that leverages a state to maintain information across time. Within the scope of deep learning, the early attempts at constructing sequence models fall under the class of non-linear RNNs ( Rumelhart & McClelland, 1987 ; Hochreiter & Schmidhuber, 1997 ) which exhibit non-linear state space dynamics. However, in recent years, models with linear state transitions have grown in popularity due to inherent trainability ( Pascanu et al., 2013 ) and efficiency ( Martin & Cundy, 2018 ) limitations in the non-linear regime. These models fall under two categories: linear systems and systems with input-varying linear operators (LIVs). We note that linear systems can further be broken down into time-invariant (LTI) and time-varying (LTV) systems, depending on whether or not the parameters of the realized recurrence change as a function of time (i.e. sequence index). This taxonomy has also been used in prior work ( Zancato et al., 2024 ) to motivate the construction of novel architectures. In modern deep learning, LTI systems ( Lindquist & Picci, 1979