Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion
Qisai Liu, Zhanhong Jiang, Md Zahid Hasan, Joshua R. Waite, Chao Liu, Aditya Balu, Soumik Sarkar · 2025
Read Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion by Qisai Liu, Zhanhong Jiang, Md Zahid Hasan, Joshua R. Waite, Chao Liu, Aditya Balu, Soumik Sarkar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08964.txt.
Introduction
Qisai Liu and co-authors published a 2025 paper on a sequence model that reads data in both directions at once. Forecasting tasks, from weather to energy demand, need models that spot patterns across time and across several data streams. Transformers do this well but slow down steeply as sequences lengthen. Classic recurrent models scale gently but usually read only forward.
Their model, BLUR, runs two linear recurrent units, one forward and one backward. Together they capture what came before and what comes after each point, while keeping the gentle scaling of recurrent designs. The system is built for fusing multiple sources at the sequence level, not just stacking their outputs. The authors present it as an efficiency play: bidirectional awareness without the transformer's quadratic cost. For long records, like sensor streams, that trade could matter more than raw size.
Catalog record
| CREATOR | Qisai Liu, Zhanhong Jiang, Md Zahid Hasan, Joshua R. Waite, Chao Liu, Aditya Balu, Soumik Sarkar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08964
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion Qisai Liu, Zhanhong Jiang, Md Zahid Hasan, Joshua R. Waite, Chao Liu, Aditya Balu, Soumik Sarkar Abstract Sequence modeling is a critical yet challenging task with wide-ranging applications, especially in time series forecasting for domains like weather prediction, temperature monitoring, and energy load forecasting. Transformers, with their attention mechanism, have emerged as state-of-the-art due to their efficient parallel training, but they suffer from quadratic time complexity, limiting their scalability for long sequences. In contrast, recurrent neural networks (RNNs) offer linear time complexity, spurring renewed interest in linear RNNs for more computationally efficient sequence modeling. In this work, we introduce BLUR (Bidirectional Linear Unit for Recurrent network), which uses forward and backward linear recurrent units (LRUs) to capture both past and future dependencies with high computational efficiency. BLUR maintains the linear time complexity of traditional RNNs, while enabling fast parallel training through LRUs. Furthermore, it offers provably stable training and strong approximation capabilities, making it highly effective for modeling long-term dependencies. Extensive experiments on sequential image and time series datasets reveal that BLUR not only surpasses transformers and traditional RNNs in accuracy but also significantly reduces computational costs, making it particularly suitable for real-world forecasting tasks. Our code is available here. Journal: Elsevier 1 Introduction In modern machine learning, sequence models have played a central role in different domains to precisely model sophisticated underlying relationships from data, such as neural machine translation [ 1 , 2 ] , time series prediction [ 3 , 4 , 5 ] , image classification and segmentation [ 6 , 7 , 8 , 9 , 10 ] , and speech recognition [ 11 , 12 , 13 ] . Among numerous models developed, two categories of models have received considerable attention and been widely adopted, which are recurrent neural networks (RNNs) [ 14 ] and Transformers [ 15 ] . Figure 1 : BLUR vs. baselines with the number of parameters against the MSE on ETTm 1 dataset with horizon=96. RNN and its variants represent the first generation of sequence models in deep learning, prior to the emergence of transformers. Despite their popularity, the vanishing and exploding gradient problems [ 16 ] have impeded the applications to modeling long-term dependencies. To mitigate this issue, LSTM [ 17 ] and GRU [ 18 ] are developed by leveraging gating mechanisms to control the flow of hidden state information along the sequence. However, they still inevitably undergo slow sequential training speed and limited capabilities in modeling long-term dependencies. In 2017, a seminal work [ 19 ] devised the attention mechanism leading to the more advanced sequence models, i.e., Transformers. They have swiftly dominated a variety of areas in sequence modeling due to outstanding abilities in parallel training and long-term dependency modeling, but with the cost of quadratic time complexity \mathcal{O}(N^{2}) , where N is assumed to be the sequence length. Unfortunately, this can make the deployment of Transformers on fairly long sequences computationally expensive. In contrast, RNN is able to achieve the linear time complexity \mathcal{O}(N) and is ideally deemed a better model for long-term sequence modeling tasks. Thus, our goal in this work is to establish a model that benefits from parallel training and maintains high computational efficiency. To this end, researchers have developed a more efficient variant of RNN, termed linear RNN , which essentially employs element-wise linear recurrence relations [ 20 ] . Motivated by this, the authors in [ 21 ] introduced the structured state-space models (S4), which achieved stunning performance on complex long-term sequence modeling tasks. Furthermore, S4 and other attempts such as DSS [ 22 ] , S4D [ 23 ] , S5 [ 24 ] , and Mamba [ 25 ] have successfully mitigated the \mathcal{O}(N^{2}) bottleneck caused by the attention layer by resorting to a hidden state with appropriate discretization schemes to model interactions between inputs. The primary advantages of these models lie in two folds. The first one is to remove nonlinearities in recurrence to retain parallel training, and the second is to assume independence between distinct hidden states to realize highly efficient updates. Another line of work on top of linear RNN is to incorporate gating mechanisms [ 26 , 27 , 28 , 29 ] , driven by LSTM and GRU, yielding considerable performance gains. Additionally, to study the difference between linear RNNs and deep state-space models, the authors in [ 30 ] tactically analyzed and ablated a series of changes to standard RNNs, including linearizing and diagonalizing the recurrence. They also introduced a novel architecture called linear recurrent unit (LRU) to fulfill parallel training and fast computation. We have also noticed that since the emergence of LRU [ 30 ] and even its precedents, such as S4, DSS, S4D, S5, and Mamba, few of them have specifically been utilized to perform the relevant timeseries tasks. For example, in [ 30 ] , all six benchmark datasets are based on images or text, although they possess temporal patterns. However, they are still naturally different from the real timeseries data that can be generated from some unknown highly nonlinear dynamical systems, which already results in the development of numerous Transformer models [ 31 , 32 , 33 ] . In addition to that, timeseries data could be so noisy that developing a precise model would become extremely challenging, such as in the cases of long-term wind power generation [ 34 ] , atmospheric circulation [ 35 ] , and ocean surface current [ 36 ] . Contributions. Regardless of the impressive performance of the aforementioned models, they are only able to make use of previous contexts. For some applications, such as speech recognition, where all the utterances are transcribed at once, it is necessary to intelligently exploit the future context as well. By doing so, we can efficiently make full use of past features (via forward pass) and future features (via backward pass) in a specific time frame. Hence, to more effectively capture the long-term dependencies in sequence modeling, we propose a novel model dubbed Bidirectional Linear Unit for Recurrent (BLUR) network, which includes forward and backward LRUs. Inheriting from LRU, BLUR still maintains both fast parallel training and inference thanks to the linear complexity \mathcal{O}(N) . BLUR also theoretically enjoys the training stability and universality. Specifically, the contributions of this work are as follows: 1. Inspired by the bidirectional recurrent models [ 37 , 38 ] , We propose the Bidirectional Linear Unit for Recurrent (BLUR) network by resorting to forward and backward LRUs. This ensures that the information from the past and future states can be captured for each time step to facilitate computationally efficient modeling of long-term dependencies. 2. We diagonalize the forward and backward recurrence matrices with the eigendecomposition and theoretically analyze the condition to provide assurance for training stability while also applying the universality theory to establish the model approximation property for BLUR. 3. To validate the proposed model, several benchmark datasets, including sequential images and timeseries are applied to show efficiency and effectiveness, with the comparison to baselines (as shown in Figure 1 ). It turns out that the bidirectional mechanism is significantly outperforming, with the cost of a slight computational time