Smooth Reading : Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks
Kai Liu, Zhan Su, Peijie Dong, Fengran Mo, Jianfei Gao, Shaoting Zhang, Kai Chen · 2025
Read Smooth Reading : Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks by Kai Liu, Zhan Su, Peijie Dong, Fengran Mo, Jianfei Gao, Shaoting Zhang, Kai Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19353.txt.
Introduction
Kai Liu and co-authors propose, in 2025, a reading strategy that lets memory-light language models handle long documents. Recurrent models run cheaply but carry a fixed-size memory, so they lag behind attention models on long texts. Earlier fixes enlarged that memory through new architectures, with limited success.
Their method, Smooth Reading, feeds the text in chunks and keeps a running summary, mimicking how people read. Memory demands drop, suiting the recurrent design. On a long-context benchmark, a recurrent model using the method moved from trailing attention models to slightly beating them, while training three times faster and answering twice as fast at long lengths.
Catalog record
| CREATOR | Kai Liu, Zhan Su, Peijie Dong, Fengran Mo, Jianfei Gao, Shaoting Zhang, Kai Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Smooth Reading : Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks Kai Liu, Zhan Su, Peijie Dong, Fengran Mo, Jianfei Gao, Shaoting Zhang, Kai Chen Abstract Recently, recurrent large language models (Recurrent LLMs) with linear computational complexity have re-emerged as efficient alternatives to self-attention-based LLMs (Self-Attention LLMs), which have quadratic complexity. However, Recurrent LLMs often underperform on long-context tasks due to their limited fixed-size memory. Previous research has primarily focused on enhancing the memory capacity of Recurrent LLMs through architectural innovations, but these approaches have not yet enabled Recurrent LLMs to match the performance of Self-Attention LLMs on long-context tasks. We argue that this limitation arises because processing the entire context at once is not well-suited for Recurrent LLMs. In this paper, we propose Smooth Reading , a chunk-wise inference method inspired by human reading strategies. Smooth Reading processes context in chunks and iteratively summarizes the contextual information, thereby reducing memory demands and making the approach more compatible with Recurrent LLMs. Our experimental results show that this method substantially narrows the performance gap between Recurrent and Self-Attention LLMs on long-context tasks, while preserving the efficiency advantages of Recurrent LLMs. Our Smooth Reading boosts SWA-3B-4k (a Recurrent LLM) from 5.68% lower to 3.61% higher performance than Self-Attention LLMs on LongBench. Besides, our method maintains the high efficiency, training 3× faster and inferring 2× faster at 64k context compared to Self-Attention LLMs. To our knowledge, this is the first work to achieve comparable performance using Recurrent LLMs compared with Self-Attention LLMs on long-context tasks. We hope our method will inspire future research in this area. To facilitate further progress, we will release code and dataset. 1 Introduction Self-attention-based large language models (Self-Attention LLMs) have achieved remarkable success across a wide range of tasks, including those requiring long-context understanding [ 3 , 18 ] . The demand for improved long-context capabilities is increasing, driven by applications such as complex reasoning [ 10 ] , embodied agents [ 43 ] , and deep research [ 45 ] . However, the Self-Attention mechanism has quadratic computational complexity and linear space requirements, severely limiting its scalability to long input sequences. In response, Recurrent LLMs have re-emerged as a promising alternative [ 27 , 15 , 40 , 39 , 34 , 38 ] . These models offer linear computational complexity and constant space usage, making them significantly more efficient than Self-Attention LLMs when processing long contexts. Despite these efficiency advantages, current Recurrent LLMs still underperform compared to Self-Attention LLMs on long-context tasks [ 37 ] . A central limitation is their fixed memory capacity [ 38 ] , which restricts their ability to retain and utilize long-range information. This leads to a trade-off: while Recurrent LLMs are better suited for processing longer contexts due to their efficiency, their actual performance on long-context tasks remains inferior. Recent research has focused on enhancing the memory capacity of Recurrent LLMs through architectural innovations [ 28 , 40 , 29 , 8 , 31 , 34 ] . For example, [ 28 ] increases memory size, while [ 40 ] introduces more expressive update rules to improve memory efficiency. Nevertheless, these architectural modifications have not yet enabled Recurrent LLMs to perform comparably with Self-Attention LLMs on long-context tasks. This situation motivates us to explore alternative strategies for improving Recurrent LLM performance in long-context scenarios. Figure 1 : Comparison of Architectures: (a) Self-Attention LLM: A new token is appended to the memory. (b) Sliding-Window LLM: A new token is appended to the memory, and the earliest token is evicted. (c) RWKV-7: “Transform”, “k”, and “v” are generated from the new token and used to update memory. The memory size of Self-Attention LLM is virtually unlimited, while the other two are fixed-size. Figure 2 : Comparison of Inference Methods for Long-context Tasks. (a) One-Step Inference: The entire context is processed at once. (b) Unsmooth Reading: The context is divided into chunks, and the model re-feeds the contextual summary at each step. (c) Smooth Reading: The model processes chunks iteratively, updating its hidden memory without re-feeding the contextual summary. We argue that a tailored inference method is essential for improving Recurrent LLMs on long-context tasks. Previous works typically relied on One-Step inference, processing the entire context in a single pass (see Figure 2 (a)). While effective for Self-Attention LLMs, which can attend to the whole sequence at once, this approach is less suitable for Recurrent LLMs, whose memory capacity is inherently limited. Inspired by the way humans read, sequentially scanning and integrating information, we introduce Smooth Reading, a novel inference strategy designed specifically for Recurrent LLMs. As illustrated in Figure 2 (c), Smooth Reading divides the input context into smaller chunks and processes them incrementally. At each step, the model reads a chunk, generates a contextual summary, and stores salient information in its hidden memory, thereby avoiding overwhelming the model with the entire long context at once. This strategy allows Recurrent LLMs to process long contexts smoothly and efficiently, providing an inference path better matched to their capabilities. Specifically, we curate a new dataset to train Recurrent LLMs specifically for the Smooth Reading procedure. Unlike architectural improvements, our method enhances Recurrent LLMs through an optimized inference process that offers explicit guidance on information importance. Notably, our technique is orthogonal to architectural modifications and can be used in conjunction with them. We evaluate our method using transformer-based LLMs with sliding-window attention (Sliding-Window LLMs) [ 4 ] and RWKV-7 [ 29 ] as representative Recurrent LLMs, and test on two long-context benchmarks: Needle-in-a-Haystack (NIAH) [ 18 ] and LongBench [ 3 ] . Our results show that Recurrent LLMs trained with Smooth Reading achieve performance comparable to, or even exceeding, that of Self-Attention LLMs on long-context tasks. Specifically, our SWA-3B-4k-SR outperforms Self-Attention LLMs by an average of 3.61% on LongBench. Moreover, Recurrent LLMs with Smooth Reading retain core advantages of recurrent architectures: (1) Length extrapolation: Recurrent LLMs are expected to generalize to contexts much longer than those seen during training. Our method can inherit the length extrapolation ability of Recurrent LLMs. For example, our SWA-3B-4k-SR, trained on sequences up to 32k tokens, can extrapolate its performance to at least 256k tokens. (2) Efficiency: Our method preserves the linear computational complexity of Recurrent LLMs, resulting in significant speed-ups. On a 64k context, our SWA-3B-4k-SR is approximately 3 \times faster in training and 2 \times faster in inference compared to the corresponding Self-Attention LLM. Our main contributions are as follows: • We propose Smooth Reading , a novel inference method for Recurrent LLMs, and curate a new dataset specifically designed to train models for this purpose. • We demonstrate that Smooth Reading significantly improves the long-context performance of Recurrent LLMs, achieving results on par with Self-Attention LLMs. To the best of our knowledge, this is the first work to close the gap using Recurrent LLMs. • We show that our approach preserves the inherent advantages of Recurrent LLMs, including superior length extrapolation and computational efficiency. 2 Related Work 2.1 Architecture of LLMs Self-Attention LLMs : Self-attention mechanisms [ 36 ] form the foundation of most widely used LLMs, such as those described in [ 10 , 32 , 1 ] . In this