Towards Robust Long-Context Understanding of Large Language Model via Active Recap Learning
Chenyu Hui \sthanks Corresponding author · 2026
Read Towards Robust Long-Context Understanding of Large Language Model via Active Recap Learning by Chenyu Hui \sthanks Corresponding author (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13734.txt.
Introduction
Chenyu Hui puts forward a way to help large language models handle very long texts in 2026. Long contexts strain models, which can lose track of early parts. Hui's method is active recap learning, or ARL. It trains models to revisit and summarize earlier content as they go.
The recap happens in two places. In training, targeted sequences teach the skill during continued pretraining. At use time, the model looks back and sums up what came before. A model that recaps well can keep the thread of long files. That skill aids law, research, and code work.
Catalog record
| CREATOR | Chenyu Hui \sthanks Corresponding author |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-13734
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Robust Long-Context Understanding of Large Language Model via Active Recap Learning Chenyu Hui \sthanks Corresponding author Abstract In this paper, we propose active recap learning (ARL), a framework for enhancing large language model (LLM) in understanding long contexts. ARL enables models to revisit and summarize earlier content through targeted sequence construction during contined pretraining and retrospective summarization at inference. First, we identify key tokens in prepared long context based on loss gaps between long and short forward contexts and find most revant preceding paragraphs, then summarize them using an LLM. Second, ARL equips models with the ability to autonomously generate and utilize these retrospective summaries during inference, thereby establishing a recursive memory mechanism across paragraphs. Experimental results show substantial gains, with ARL achieving a 26.8% improvement on RULER and a 9.44% improvement on LongBench. Overall, ARL offers a simple yet effective continued pretraining-based approach to strengthen long-context understanding, advancing scalable memory augmentation in LLM.Code is aviliable at CODE † † address: 1 School of Microelectronics Xi’an Jiaotong University Xi’an, China Figure 1: Overview framework of our proposed ARL with a chunk generating recap example. 1 Introduction Long-context understanding is a core capability of large language model (LLM), reflected not only in their ability to process extended text sequences but also in their capacity to comprehend, reason over, and leverage complex information within them. Robust long-context understanding is therefore critical for applying LLM to real-world scenarios such as legal documents [ 5 ] , academic manuscripts [ 12 ] , financial reports [ 9 ] , and large codebases [ 11 ] . However, in practice, issues such as performance degradation under ultra-long inputs, memory constraints, and computational scalability often lead to suboptimal outcomes. Current research on enhancing long-context understanding can be broadly categorized into three directions: length extrapolation, efficient attention, and context compression. The first line of work, which extends positional encoding or employs continued pretraining, often yields limited improvements while incurring high computational costs [ 16 ] . The second line, which introduces linear or kernelized attention mechanisms, typically requires training models from scratch and encounters challenges in parallelization and scalability [ 8 , 17 ] . The third line, which relies on token compression or external memory modules, tends to disrupt the generation process and exhibits limited compatibility with mainstream frameworks [ 14 ] . Despite these advances, existing state-of-the-art methods have yet to achieve a balance of efficiency, generalization, and usability, leaving LLM performance on long-text tasks far from satisfactory. To address these challenges, we propose active recap learning (ARL), a simple yet effective framework that strengthens a model’s ability to recall and exploit relevant context from earlier portions of long texts. As illustrated in Fig. 1 , ARL neither increases model capacity nor compresses information. Instead, it enables the model to perform active recapping during both continued pretraining and inference, thereby establishing a recursive memory mechanism. Concretely, we first construct novel pretraining sequences by identifying key tokens whose predictions are highly sensitive to the truncation of preceding context. For these tokens, we extract the most informative early segments and summarize them using an LLM. The resulting summaries are reinserted into the long text with special tags, allowing the model to learn context-recall patterns. During inference, the input text is segmented, and the same tagging mechanism prompts the model to generate a summary for each segment. These summaries are then propagated segment by segment, simulating a memory function that preserves coherence and consistency throughout long texts. The main contributions of this paper can be summarized as follows: • We introduce the long-short gap metric for high-specificity context tagging, enabling precise identification of critical information. • We design a two-stage contextual extraction approach to identify optimal segments and refine LLM processing of long inputs. • We develop an inference-time recap agent capable of generating local summaries and synthesizing global context, thereby maximizing the effectiveness of the recap mechanism. 2 Proposed Method We propose active recap learning (ARL), a framework designed to improve long-range context modeling in large language models. ARL operates in three stages: (1) identifying key tokens using the long-short gap metric, (2) retrieving the most relevant earlier segments via loss-based retrieval, and (3) refining these segments into concise summaries. The augmented sequences are then used for continued pretraining, while a lightweight recap agent applies the same mechanism at inference to segment inputs and propagate summaries, enabling effective contextual recall across long texts. 2.1 Recap Supervision To enhance long-context modeling, we first identify tokens that strongly depend on distant information. We introduce the long-short gap (LSG) metric, which measures a token’s reliance on long-range context by comparing prediction probabilities under two settings: (1) the full context and (2) a truncated short context. Formally, for token x_{i} : I_{\mathrm{LSG}}(x_{i},\theta)=\frac{P_{\theta}(x_{i}\mid\boldsymbol{l}_{i})}{P_{\theta}(x_{i}\mid\boldsymbol{s}_{i})}, (1) where P_{\theta}(x_{i}\mid\boldsymbol{l}_{i}) and P_{\theta}(x_{i}\mid\boldsymbol{s}_{i}) denote the model’s predicted probabilities with long and short contexts, respectively. A higher score indicates stronger dependence on earlier content.We then select the Top-K tokens with the highest LSG scores as key tokens, which serve as anchors for retrieving relevant segments from preceding text. Given the key tokens identified by the LSG metric, we aim to locate the most informative preceding segment that enhances the prediction of each key token. Candidate segments are extracted from earlier parts of the document via a sliding window, aligned to sentence boundaries for coherence. For a key token x_{i} , the optimal segment c^{*} is defined as: c^{*}=argmax_{c\in\mathcal{C}}P_{\theta}\big(x_{i}\mid\text{Insert}(c,\mathcal{D},i)\big), (2) where \text{Insert}(c,\mathcal{D},i) denotes inserting segment c before x_{i} in document \mathcal{D} . This selection ensures that the chosen context maximally improves the model’s confidence in predicting the target token, thereby reinforcing long-range dependency modeling. The primary procedures of finding the most relevant remote previous text fragment are shown in Algorithm 1 . Algorithm 1 : Mining Most Relevant Segments 1: 2: text : Input document 3: top_k : Number of key tokens to select 4: window_size : Sliding window size (in characters) 5: step_size : Step size for window traversal 6: 7: A list of refined recap segments 8: \text{key\_tokens}\leftarrow\text{SelectTopKTokens}(text,top\_k) ; 9: for x_{i}\in\text{key\_tokens} do 10: best\_context\leftarrow\text{None},\ best\_score\leftarrow 0 ; 11: for start=0 to len(text) step step\_size do 12: c\leftarrow\text{text}[start:start+window\_size] ; 13: text^{\prime}\leftarrow\text{Insert}(c,text,i) ; 14: p\leftarrow P_{\theta}(x_{i}\mid text^{\prime}) ; 15: if p>best\_score then 16: best\_context\leftarrow c,\ best\_score\leftarrow p ; 17: end if 18: end for 19: refined\leftarrow\text{LLM.Rephrase}(best\_context) ; 20: \text{recaps.append}(\texttt{<re>}+refined+\texttt{</re>}) ; 21: end for 22: return recaps After selecting the most relevant segment, we refine it using a LLM to enhance coherence and semantic clarity. The refined segment, termed a ”recap”, is enclosed with special tags <\texttt{re}> and </\texttt{re}> as explicit supervision signals. These recaps are further summarized into compact forms. In this process, the designed prompt emphasize that the output should be concise (typically 5 or 6 sentences) while covering all essential information. 2.2 Recap Agent After continued pretraining on mined sequences. To leverage recap segments at inference, we introduce the recap agent. The input document is divided into non-overlapping chunks, each appended with a <\texttt{re}> token to prompt the model to summarize key information. When the accumulated recap contents exceeds the threshold N , the earliest portions are summarized to prevent uncontrolled context growth while retaining salient historical information. This dynamic truncation ensures effective long-range reasoning