Analysis of Long Range Dependency Understanding in State Space Models
Srividya Ravikumar, Abhinav Anand, Shweta Verma, Mira Mezini · 2026
Read Analysis of Long Range Dependency Understanding in State Space Models by Srividya Ravikumar, Abhinav Anand, Shweta Verma, Mira Mezini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13048.txt.
Introduction
Srividya Ravikumar, Abhinav Anand, Shweta Verma, and Mira Mezini probe how state space models hold long range links in data. Such models score well on long chain tasks. Most past work, though, judged them by score alone and not by how they work inside.
The team runs what they call the first full study of kernel sense in one such model, S4D. They train it on a real task, the hunt for flaws in source code that could aid attacks. They then read the trained kernels to see what spans the model keeps or drops. For builders, the study shifts talk from raw score to clear cause. It shows which long links the model truly uses.
Catalog record
| CREATOR | Srividya Ravikumar, Abhinav Anand, Shweta Verma, Mira Mezini |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13048
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Analysis of Long Range Dependency Understanding in State Space Models Srividya Ravikumar, Abhinav Anand, Shweta Verma, Mira Mezini Abstract Although state-space models (SSMs) have demonstrated strong performance on long-sequence benchmarks, most research has emphasized predictive accuracy rather than interpretability. In this work, we present the first systematic kernel interpretability study of the diagonalized state-space model (S4D) trained on a real-world task (vulnerability detection in source code). Through time and frequency domain analysis of the S4D kernel, we show that the long-range modeling capability of S4D varies significantly under different model architectures, affecting model performance. For instance, we show that the depending on the architecture, S4D kernel can behave as low-pass, band-pass or high-pass filter. The insights from our analysis can guide future work in designing better S4D-based models. † † address: ∗ TU Darmstadt † Hessian Center for Artificial Intelligence, Darmstadt, Germany ‡ National Research Center for Applied Cybersecurity ATHENE 1 Introduction Structured State Space Models (SSMs) [ 1 ] have recently been introduced as efficient architectures for sequence modeling, showing competitive results in language, time series, and program analysis tasks [ 2 ] . By parameterizing input–output mappings through long convolutional kernels, they provide a scalable way to capture long-range dependencies. The original S4 model demonstrated this potential on long-sequence benchmarks, positioning SSMs as strong alternatives to RNNs and transformers. The diagonal variant of SSM, S4D [ 3 ] , further improved computational efficiency and stability by simplifying kernel generation, making it more practical for large-scale applications. Recent work on understanding source code has explored both transformer-based approaches and state-space variants, with CodeSSM [ 2 ] indicating that SSMs can effectively model the syntactic and semantic structure of code. Despite these advances, the mechanisms underlying the behavior of SSM remain insufficiently understood. Previous research has examined stability and generalization properties of SSMs (see Section 4 ), but the practical implications for kernel behavior are less clear. In particular, it is not well established whether kernels emphasize local transitions, long-range dependencies, or specific spectral bands, nor how these properties vary across architectures. This lack of interpretability limits our ability to connect theoretical predictions with empirical outcomes. In this work, we study the interpretability of SSM kernels using the diagonalized variant S4D, which provides computational efficiency and stable kernel generation while preserving the modeling capacity of SSMs [ 3 ] . We analyze CNN–S4D hybrid architectures in both the time and frequency domains to examine how architectural choices shape kernel responses and influence the modeling of dependencies. For our study, we used single-layer models to maintain architectural simplicity and interpretability, enabling detailed analysis of how individual kernels respond to specific code patterns. We trained the models on a real-world task (vulnerability detection in source code) instead of synthetic datasets used in previous works on SSM analysis. Single layer 1D CNNs have already been shown to perform well on this task. So, we use a single-layer CNN baseline to evaluate the performance of different S4D-based models. Our observations depict that the standalone S4D model performs worse than CNN. Consistent with this observation, our analysis of learned S4D kernel reveals that standalone S4D model only learns short-range dependencies, despite its theoretical long-range modeling capacities [ 1 , 4 ] . On the other hand, the S4D model preceded by a state memory replay [ 5 ] block performs the best and also shows long-range understanding in our analysis. 2 METHODOLOGY In this section, we explain the different model architectures we have trained and the proposed methodology to analyze the behavior of the SSM kernel. 2.1 Model architectures We implemented six model architectures to evaluate performance and study kernel behavior in vulnerability detection. All models share a common architecture (Figure 1 ), where tokenized source code is mapped to dense embeddings and processed through a feature extraction block. The outputs are passed through ReLU activation, max pooling, concatenation, dropout, and a fully connected layer for binary classification. The feature extraction block is the only component that varies between models. Figure 1: Common model architecture used across all experiments. The Feature extraction block represents different feature transformation strategies across six model variants. Convolution 1D baseline model: A single one-dimensional convolutional layer that applies multiple kernel sizes in parallel to extract local features similar to CNN architectures used in sentence classification tasks [ 6 ] . Depthwise separable convolution model: The model factors the standard convolution into two successive stages: a convolution in depth, where a single filter is applied per channel to extract the localized structure, and a pointwise (1×1) convolution, which mixes information between channels significantly reducing the number of parameters and computational cost while maintaining predictive performance [ 7 ] . S4D (standalone) model: In this architecture the feature extraction block contains a single Structured State Space (S4D) layer. The S4D efficiently computes parameterized kernels in the frequency domain using FFTs. Depthwise separable with S4D model: This hybrid design first applies a depth-wise convolution for local feature extraction, followed by an S4D layer with global context modeling, and finally a point-wise convolution to integrate information across channels. This model was designed to illustrate how SSMs and CNNs can complement each other in capturing both structural and semantic cues in code. Conv1D + S4D model: This hybrid model extends the Conv1D baseline by stacking an S4D layer after the convolutional layer. The convolution layer captures fine-grained local features, while the subsequent S4D module models broader temporal dependencies across the sequence. State Memory Replay(SMR) + S4D model: The final variant introduces a State Memory Replay (SMR) mechanism [ 5 ] wherein the outputs of the Conv1D layer are multiplied element-wise with the original embedding representations before being passed to the S4D layer, as shown in Figure 2 . This fusion mechanism improves the ability of the model to recall contextual memory while preserving local structure. Figure 2: Feature extraction block for SMR + S4D model. 2.2 Training We train all models on the ReVeal dataset [ 8 ] , a real-world benchmark for vulnerability detection. The dataset was constructed from two large-scale open source projects, the Linux Debian kernel and Chromium, covering diverse security issues across operating systems and browsers. It contains 22,734 functions, of which 2,240 are vulnerable and 20,494 are nonvulnerable, formulated as a binary classification task. To address class imbalance, we used BCEWithLogitsLoss with class weights. All models were trained using Adam Optimizer (learning rate = 1\times 10^{-3} ), with dropout 0.5, sequence length capped at 256 tokens, batch size 64, and 10 training epochs. Early stopping was not applied to maintain consistent training duration across runs. The data set was divided into 80% training, 10% validation and 10% testing, and all experiments were implemented in PyTorch and executed on CUDA 11.8. 2.3 Kernel analysis To interpret the internal behavior of the trained models, we performed a kernel-level analysis in both the time domain and the frequency domain. 2.3.1 Time-domain analysis The convolutional and S4D kernels were extracted after training and were visualized in the temporal domain. We evaluated two key properties: • Peak sensitivity: maximum positive and negative amplitudes, indicating the strongest excitations of the kernel. • Transition sharpness: first-order differences that highlight abrupt changes in kernel response. This analysis reveals