Timesteps of Mamba Align with Human Reading Times

Yuji Yamamoto Shinnosuke Isono Yoshinobu Kawahara Sho Yokoi · 2026

Read Timesteps of Mamba Align with Human Reading Times by Yuji Yamamoto Shinnosuke Isono Yoshinobu Kawahara Sho Yokoi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29904.txt.

Introduction

Yamamoto and three colleagues compare the internal timing of Mamba, a state-space language model, with human reading times. In Mamba each word triggers a state update whose step size, the discretization timestep, is set dynamically by the input.

Using a naturalistic dataset of people reading text, the team shows that Mamba's per-word timestep significantly predicts how long humans spend on each word. The link survives even after controlling for known predictors such as GPT-2 surprisal, how unexpected a word is. The authors argue that Mamba's layered memory, which weighs short-term and long-term retention differently at each layer, offers a new lens on how humans process language in real time.

Catalog record

CREATORYuji Yamamoto Shinnosuke Isono Yoshinobu Kawahara Sho Yokoi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29904

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Timesteps of Mamba Align with Human Reading Times Yuji Yamamoto Shinnosuke Isono Yoshinobu Kawahara Sho Yokoi Abstract This study demonstrates an alignment of per-word processing time in a popular state-space language model Mamba and human readers. In Mamba, the recurrent state transition at each layer conceptually takes some duration of time, the discretization timestep Δ t \Delta_{t} , determined dynamically in response to the input. Using a naturalistic reading dataset, we show that the per-word timestep from Mamba is a significant predictor of human reading times, and remains significant even when known predictors such as GPT-2 surprisal are controlled for. We further suggest, through formal analysis of Mamba’s architecture and internal dynamics, that Mamba can serve as a new, valuable lens to look at human real-time language processing with ever-updated memory, because it allows us to look at how each module (layer) weighs short- and long-term information retention, and how noise may interact with dynamic, continuous memory representation. Code is available online. 1 1 1 Code available at https://osf.io/vnw5e/overview?view_only=93ad704fc6ea44438f3d3538b4b682eb * * footnotetext: Equal contribution 1 Introduction Human language inherently unfolds over time. In the rise of fluent artificial large language models (LMs), it has become interesting to ask how time may be represented in (unidirectional variants of) them, and how their time is related to temporal dynamics of language processing in humans. Mamba Gu and Dao 2024 , a popular architecture among LMs based on a state-space model (SSM), is interesting in this respect, since it has a notion of internal timesteps, which modulates state transitions in response to input. SSMs represent sequential data as continuous state transitions and are characterized by their ability to handle long-range dependencies in a stable manner. Among them, Mamba is designed as a selective SSM that dynamically modulates state transitions by changing the size of timesteps in response to the input, offering greater flexibility than other SSMs that assume fixed transition dynamics. As illustrated in Figure 1 , this property enables the model to immediately reflect changes of context and importance of particular input in its internal states, allowing adaptive representation learning even for data with complex and non-stationary structures. Figure 1: Illustration of discretization of a continuous-time state transition h(t) using the input-dependent timestep \Delta_{t} . Figure 2: Discretization timesteps \bar{\Delta}_{t} of each of the 24 layers of Mamba-130M, in response to the first five sentences from the Natural Stories Corpus. “★” indicates the beginning of a sentence. In this study, we demonstrate a word-level alignment between Mamba’s internal timesteps and human processing time. After establishing that Mamba’s internal timesteps can be seen as processing time (§ 3 ), we show that the sizes of the timesteps that Mamba assigns to each word in a naturalistic corpus is predictive of the time that human readers spend on each word (§ 4 ). The performance of the best-performing layer is comparable to that of GPT-2 surprisal, a variable known to be a powerful predictor of human reading times, and remains significant even when GPT-2 surprisal and other known predictors are controlled for. We also conducted formal analyses looking into how Mamba’s architecture and internal dynamics process language over time, and argue that the model can serve as a new, valuable lens to look into human real-time language processing with ever-updated memory (§ 5 ). Specifically, we observe that the model’s transition matrix eigenspectrum specifies a different degree of memory retention ability for each layer, enabling the entire model to process both short- and long-term information. We also theoretically show that the size of the timesteps corresponds to uncertainty in state transition, and suggest that this offers a new insight into noisy language processing. 2 Preliminaries 2.1 Mamba We provide an overview of Mamba; for further details, see Gu and Dao 2024 . Mamba is a popular state-space model (SSM)-based language model. Unlike Transformers, SSMs do not process the entire sequence simultaneously but instead handle input and output recursively and sequentially on a word-by-word basis, in a similar fashion to RNNs and human readers. Because SSMs are recursive, a naive implementation would require computation time proportional to the input sequence length. However, since the hidden-state update equations of SSMs contain no nonlinearities, the model can be accelerated using parallel scan algorithms Blelloch 1990 , making large-scale language model training feasible. Mamba, one of the representative state-space models, as well as Linear Attention Yang and Zhang 2024 models that also employ linear state updates, are employed in the development of highly efficient large language models Zuo et al. 2024 ; Preferred Networks et al. 2025 ; Kimi Team et al. 2025 ; Blakeman et al. 2025 . Mamba has multiple layers (for example, the Mamba-130M model used in the current experiment consists of 24 layers 2 2 2 https://huggingface.co/state-spaces/mamba-130m-hf ). Each layer incorporates a discretized form of a state-space model. While state-space models are defined as differential equations in continuous time, they are approximated as difference equations in discrete time when applied to real data, particularly discrete data such as language (Figure 1 ). In standard discretization procedures, a constant discretization timestep \Delta is used regardless of the time index t ; in contrast, Mamba assigns an input-dependent discretization timestep \Delta_{t} to each timestep t . The discretization timestep \Delta_{t} acts as a gate in each layer of Mamba, governing memory retention and forgetting. Below, we formally explain this by referring to the specific update equations of Mamba. Given an input \bm{x}_{t}\in\mathbb{R}^{d} , Mamba applies the following equations independently to each element x_{ti} of \bm{x}_{t} in parallel d times, and outputs the vector \bm{y}_{t}\in\mathbb{R}^{d} obtained by collecting the resulting outputs y_{ti} . Mamba consists of two equations: one updates a component called the hidden state \bm{h}_{t}\in\mathbb{R}^{n} , which corresponds to memory, using the input x_{ti} , and the other generates the output y_{ti} from the hidden state \bm{h}_{t} : \displaystyle\bm{h}_{t} \displaystyle=\overline{A}_{t}\bm{h}_{t-1}+\overline{B}_{t}x_{ti} (1) \displaystyle y_{ti} \displaystyle=C_{t}\bm{h}_{t}+Dx_{ti}. (2) Here, the state transition matrix \overline{A}_{t}\in\mathbb{R}^{n\times n} and input matrix \overline{B}_{t}\in\mathbb{R}^{n\times 1} , which represent the weights for memory and input in the state update equation, are defined as follows with an input-dependent discretization timestep \Delta_{t}~\in~(0,~\infty) : \displaystyle\overline{A}_{t} \displaystyle=\exp(\Delta_{t}A)~\left(=\exp(A)^{\Delta_{t}}\right) (3) \displaystyle\overline{B}_{t} \displaystyle=\Delta_{t}W_{B}\bm{x}_{t} (4) \displaystyle\Delta_{t} \displaystyle=\log(1+\exp(W_{\Delta}\bm{x}_{t}+\bm{b}_{\Delta})). (5) As can be seen from the above equations, the discretization timestep \Delta_{t} plays an important role in modulating the weight coefficients for memory and input. For example, when \Delta_{t} is large, the elements of the matrix \overline{A}_{t} approach zero, 3 3 3 In Mamba, the transition matrix \exp(A) is implemented to be diagonal, with elements constrained to lie within the range (0,1] . thereby reducing the influence of the previous memory \bm{h}_{t-1} . At the same time, the absolute values of the input weight coefficients \overline{B}_{t} increase, strengthening the influence of the input \bm{x}_{t} . 4 4 4 Conversely, when \Delta_{t} is small, that is, close to zero, the elements of \overline{A}_{t} approach one and those of \overline{B}_{t} approach zero, preserving the previous memory \bm{h}_{t-1} while discarding the input \bm{x}_{t} . 2.2 Reading Time Modeling Reading time modeling is a research approach that aims to quantitatively identify factors contributing to cognitive load during sentence comprehension by explaining variations in the reading time that readers spend reading each word in a text. Reading time is

Read the complete text