pLSTM: parallelizable Linear Source Transition Mark networks

Korbinian Pöppel, Richard Freinschlag, Thomas Schmied, Wei Lin, Sepp Hochreiter · 2025

Read pLSTM: parallelizable Linear Source Transition Mark networks by Korbinian Pöppel, Richard Freinschlag, Thomas Schmied, Wei Lin, Sepp Hochreiter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11997.txt.

Introduction

Korbinian Pöppel and co-authors add a new member in 2025 to a family of fast neural networks that work step by step. Such networks have lately rivaled the transformer on language tasks. Their weak point is shape: most handle flat chains of data, while images, trees, and webs have a richer layout that must be forced into a chain.

Older networks built for rich layouts exist, but they run one step at a time and are slow. The new design borrows their layout skill and adds gates that let steps run in parallel, like the fast models do. It works on the line graph form of a general web of linked steps. That opens fast training to data shaped as grids, trees, and webs. The authors set out the design and how it can be trained at scale.

Catalog record

CREATORKorbinian Pöppel, Richard Freinschlag, Thomas Schmied, Wei Lin, Sepp Hochreiter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-11997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

pLSTM: parallelizable Linear Source Transition Mark networks Korbinian Pöppel, Richard Freinschlag, Thomas Schmied, Wei Lin, Sepp Hochreiter Abstract Modern recurrent architectures, such as xLSTM and Mamba, have recently challenged the Transformer in language modeling. However, their structure constrains their applicability to sequences only or requires processing multi-dimensional data structures, such as images or molecular graphs, in a pre-defined sequential order. In contrast, Multi-Dimensional RNNs (MDRNNs) are well suited for data with a higher level structure, like 2D grids, trees, and directed acyclic graphs (DAGs). In this work, we extend the notion of multi-dimensionality to linear RNNs. We introduce parallelizable Linear Source Transition Mark networks (pLSTMs) using Source, Transition, and Mark gates that act on the linegraph of a general DAG. This enables parallelization in analogy to parallel associative scans and the chunkwise-recurrent form of sequential linear RNNs, but for DAGs. For regular grids (1D and 2D), like images, this scheme can be efficiently implemented using einsum operations, concatenations, and padding in logarithmic time. pLSTMs tackle the vanishing/exploding activation/gradient problem for long distances in DAGs via two distinct modes: a directed propagation mode (P-mode) and a diffusive distribution mode (D-mode). To showcase the long-range capabilities of pLSTM, we introduce arrow-pointing extrapolation as a synthetic computer vision task that contains long-distance directional information. We demonstrate that pLSTMs generalize well to larger image sizes, whereas Transformers struggle to extrapolate. On established molecular graph and computer vision benchmarks, pLSTMs also show strong performance. Code and Datasets are available at: https://github.com/ml-jku/plstm_experiments . 1 Introduction Linear RNNs such as DeltaNet ( Schlag et al. 2021 ) , Gated Linear Attention ( Yang et al. 2023 ) , Mamba ( Gu and Dao 2023 ; Dao and Gu 2024 ) , and xLSTM (mLSTM) ( Beck et al. 2025b ) have recently evolved as a powerful alternative to the Transformer ( Vaswani et al. 2017 ) . In addition to sequence-parallelizable training, such modern recurrent architectures are more efficient at inference time than Transformers. In contrast to classical RNNs, they come with associative memory expansion, enabling more performant sequence modeling. However, these modern recurrent architectures are inherently limited to sequences. While linear RNNs have shown good performance for multi-dimensional data such as images, they enforce a pre-defined sequential traversal structure ( Zhu et al. 2024 ; Alkin et al. 2025 ) . To illustrate, for images, this corresponds to processing patches in a scanline form. However, this is problematic as, within one layer, even (vertical) neighbors in a patch grid are only distantly related, in the (horizontal line-wise) processing. Even when switching between a line- and column-wise order across layers, diagonal relationships are not covered sufficiently. This mismatch of short-range spatial and long-distance relations in a certain path with precise positioning requirements leads to credit assignment problems ( Minsky 1961 ; Bengio and Frasconi 1993 ; Schmidhuber 2015 ) and suffers from vanishing activations/gradients ( Hochreiter 1991 ; Bengio et al. 1994 ; Hochreiter et al. 2001 ; Hochreiter and Schmidhuber 1997 ; Pascanu et al. 2013 ) . Figure 1 : Illustration of the receptive fields induced by pLSTM and related architectures (for a single layer). CNNs are locally restricted while ViTs have a global receptive field. Modern recurrent architectures, such as ViM, traverse the 2D grid sequentially. pLSTM effectively extends the receptive field via its combination of D-mode and P-mode. Multi-dimensional RNNs (MDRNNs) ( Graves et al. 2007 ) have demonstrated how non-linear RNNs like LSTMs ( Hochreiter 1991 ; Hochreiter and Schmidhuber 1997 ) can be extended to multi-dimensional data, such as images ( Graves et al. 2007 ; van den Oord et al. 2016 ) , trees ( Tai et al. 2015 ) , and DAGs ( Peng et al. 2017 ) . In this work, we translate the findings of MDRNNs to linear RNNs and introduce parallelizable Linear Source Transition Mark Networks (pLSTM). To achieve this, we first translate their input, forget, and output gates to Source, Transition, and Mark gates, respectively. This is to make the resulting structure applicable to general DAGs. Second, we derive a parallelization scheme enabled by the previous adaptation, leading to higher-order Source, Transition, and Mark terms - forming a parallel associative scan in multiple dimensions and DAGs. This also allows for a chunkwise-parallel formulation known from linear RNNs ( Yang et al. 2023 ; Beck et al. 2025a ) , using the parallelization up to a certain level and the recurrent option between chunks/patches. Third, we derive two stabilization options, the P-mode for directional propagation and the D-mode for undirected global distribution. In Figure 1 , we illustrate how the two modes affect the receptive field of pLSTM compared to other linear RNNs. We use pLSTM with MLP layers in a residual pre-LayerNorm backbone, resembling a Transformer with replaced multi-head attention. To showcase the long-range capabilities of pLSTM in multiple dimensions, we first introduce the synthetic arrow-pointing extrapolation task (see Figure 2 ). As the directional information is not tied to only testing horizontal and vertical cases separately, a certain line-wise or column-wise ordering (even in alternation) cannot capture the directional information in the sequential RNN transitions, at least for a limited number of layers. Importantly, pLSTM generalizes well to increasing image resolutions compared to the Transformer (see Section 5.1 ). Moreover, we demonstrate the efficacy of pLSTM on established computer-vision and graph benchmarks. Experiments on ImageNet-1k (see Section 5.2 ) and on molecular graphs (see Section 5.4 ) show promising results compared to baselines and good scaling behavior to larger model sizes. In summary, we make the following contributions : • We translate the findings of classic MDRNNs to linear RNNs and introduce pLSTM, which comes with adapted gates and a scalable chunkwise-parallel formulation. • We formally derive the general stabilization of pLSTMs for long-range propagation on general DAGs (including images), establishing the P- and D-mode cases. • We introduce the synthetic Arrow Pointing Task to highlight the theoretical advantage of pLSTMs, in which pLSTM shows strong extrapolation abilities, and provide a highly scalable implementation of pLSTM. 2 Related work Modern Recurrent Architectures This work presents a new form of linear RNNs, where DeltaNet ( Schlag et al. 2021 ; Yang et al. 2024 ) , LRU ( Orvieto et al. 2023 ) , GLA ( Yang et al. 2023 ) , Mamba ( Gu and Dao 2023 ; Dao and Gu 2024 ) , and xLSTM ( Beck et al. 2025b ) (in the mLSTM form) have shown their effectiveness on sequence modeling in the language domain. Recently, this line of work has been complemented by TTT ( Sun et al. 2024 ) , Titans ( Behrouz et al. 2024 ) , and DeltaProduct ( Siems et al. 2025 ) , which motivate this structure as a gradient-based optimization in context, in line with early work on Fast-Weight Programmers ( Schlag et al. 2021 ) . Due to their ability for parallelization during training, modern recurrent architectures scale to large-scale datasets similar to Transformers. Moreover, they come with efficient inference, which is attractive for real-world applications ( Schmidinger et al. 2024 ; Schiff et al. 2024 ; Schmied et al. 2024 ) . Figure 2 : Illustration of the Arrow pointing task . The

Read the complete text