EndPrompt: Efficient Long-Context Extension via Terminal Anchoring
Han Tian 1,2∗ Luxuan Chen 3,2∗ Xinran Chen 2 Rui Kong 2 Fang Wang 2 Jiamin Chen 2 Jinman Zhao 2 Yuchen Li 2† Jiashu Zhao 2 Shuaiqiang Wang 2 Haoyi Xiong 4 Linghe Kong 3 Dawei Yin 2† 1 Nankai University 2 Baidu Inc. 3 Shanghai Jiao Tong University 4 Independent Researcher [email protected], [email protected], [email protected] · 2026
Read EndPrompt: Efficient Long-Context Extension via Terminal Anchoring by Han Tian 1,2∗ Luxuan Chen 3,2∗ Xinran Chen 2 Rui Kong 2 Fang Wang 2 Jiamin Chen 2 Jinman Zhao 2 Yuchen Li 2† Jiashu Zhao 2 Shuaiqiang Wang 2 Haoyi Xiong 4 Linghe Kong 3 Dawei Yin 2† 1 Nankai University 2 Baidu Inc. 3 Shanghai Jiao Tong University 4 Independent Researcher [email protected], [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14589.txt.
Introduction
Tian and colleagues stretch a language model's context window using only short training sequences. Their EndPrompt keeps the short context intact as a first segment. It appends a brief terminal prompt, forming a second segment, with positional indices near the target length. This exposes long-range distances without full-length inputs. Applied to LLaMA-family models from eight thousand to sixty-four thousand tokens, it is claimed to top rival methods on LongBench.
Catalog record
| CREATOR | Han Tian 1,2∗ Luxuan Chen 3,2∗ Xinran Chen 2 Rui Kong 2 Fang Wang 2 Jiamin Chen 2 Jinman Zhao 2 Yuchen Li 2† Jiashu Zhao 2 Shuaiqiang Wang 2 Haoyi Xiong 4 Linghe Kong 3 Dawei Yin 2† 1 Nankai University 2 Baidu Inc. 3 Shanghai Jiao Tong University 4 Independent Researcher [email protected], [email protected], [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41090 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14589
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EndPrompt: Efficient Long-Context Extension via Terminal Anchoring Han Tian 1,2∗ Luxuan Chen 3,2∗ Xinran Chen 2 Rui Kong 2 Fang Wang 2 Jiamin Chen 2 Jinman Zhao 2 Yuchen Li 2† Jiashu Zhao 2 Shuaiqiang Wang 2 Haoyi Xiong 4 Linghe Kong 3 Dawei Yin 2† 1 Nankai University 2 Baidu Inc. 3 Shanghai Jiao Tong University 4 Independent Researcher [email protected], [email protected], [email protected] Abstract Extending the context window of large language models typically requires training on sequences at the target length, incurring quadratic memory and computational costs that make long-context adaptation expensive and difficult to reproduce. We propose EndPrompt , a method that achieves effective context extension using only short training sequences. The core insight is that exposing a model to long-range relative positional distances does not require constructing full-length inputs: we preserve the original short context as an intact first segment and append a brief terminal prompt as a second segment, assigning it positional indices near the target context length. This two-segment construction introduces both local and long-range relative distances within a short physical sequence while maintaining the semantic continuity of the training text–a property absent in chunk-based simulation approaches that split contiguous context. We provide a theoretical analysis grounded in Rotary Position Embedding and the Bernstein inequality, showing that position interpolation induces a rigorous smoothness constraint over the attention function, with shared Transformer parameters further suppressing unstable extrapolation to unobserved intermediate distances. Applied to LLaMA-family models extending the context window from 8K to 64K, EndPrompt achieves an average RULER score of 76.03 and the highest average on LongBench, surpassing LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23) while requiring substantially less computation. These results demonstrate that long-context generalization can be induced from sparse positional supervision, challenging the prevailing assumption that dense long-sequence training is necessary for reliable context-window extension. The code is available at https://github.com/clx1415926/EndPrompt . † † footnotetext: ∗ Co-first authors with equal contributions. † † footnotetext: † Corresponding author. 1 Introduction Large language models are the foundation of modern natural language processing and a central interface for complex reasoning. However, their reliable maximum context length constrains utility. Applications such as long-document question answering [ 13 , 23 ] , repository-level code understanding [ 22 , 16 ] , legal and scientific document analysis [ 4 ] , personalized assistants, and extensive evidence retrieval require reasoning over inputs exceeding pretraining context windows. Here, longer contexts are effective only if models preserve local coherence and establish reliable interactions between distant tokens. Consequently, extending pretrained context windows is a central problem in language model adaptation. Continuing to train models on long sequences at the target context length is a straightforward but costly solution. Collecting high-quality long-form corpora is difficult, and training increases memory consumption and runtime due to the quadratic scaling of attention computation [ 30 ] . While efficient implementations like FlashAttention [ 12 ] and distributed systems mitigate this burden, full-length fine-tuning remains expensive. To reduce costs, recent methods explore position interpolation [ 8 , 26 , 6 , 14 ] , sparse attention [ 10 ] , sliding-window attention [ 3 , 32 , 17 ] , low-rank adaptation [ 20 ] , and simulated long-context training [ 35 , 1 ] . However, these approaches often introduce limitations, such as requiring substantial long-sequence training, altering the structure of attention, or chunking text in ways that damage semantic continuity. Thus, it remains unclear whether context extension requires dense supervision at the target length or if a sparse set of structured positional signals is sufficient. This paper investigates whether models must observe full-length sequences to acquire long-context capabilities. We explore this question through the lens of positional generalization. In models utilizing Rotary Position Embedding [ 29 ] , attention scores depend on both token content and relative positional distance. While existing methods assume reliable extrapolation requires exposure to dense relative distances during training, we demonstrate that effective long-context behavior emerges from sparse training signals. Specifically, models trained on short sequences can receive supervision for long-range positions if the examples preserve semantic coherence and provide stable anchors for distant positions. This finding reframes context adaptation as the design of informative positional supervision rather than merely increasing physical sequence length. We propose EndPrompt , an efficient context-extension method utilizing positional index manipulation and an appended end prompt. We retain the original short context as the first segment and append a terminal prompt as the second. The first segment receives local positional indices, and the second receives indices near the target maximum context length. This configuration generates both local and long-range relative distances within a short physical sequence. Unlike chunk-based methods [ 35 ] , our approach avoids splitting contiguous text, thereby preserving semantic integrity while exposing models to long-distance positional patterns. The end prompt functions as a stable terminal anchor. Experiments indicate robustness across various prompt formulations, suggesting efficacy derives from the terminal cue’s structural position rather than memorizing specific tokens. This design exposes long-range positional interactions without compromising the quality of the short-context training signal. We analyze how sparse supervision supports long-context generalization. Under Rotary Position Embedding [ 29 ] , attention scores act as a sum of sinusoidal components over relative distances, featuring content-dependent amplitudes and phases. Position interpolation [ 8 ] reduces effective positional frequencies, constraining the attention score’s variation rate and curvature across the distance dimension. This reduction induces a smoothness bias over unobserved intermediate distances. Furthermore, Transformers do not learn independent parameters for each distance; the same query and key projections support local and long-distance behavior. Consequently, sparse long-range supervision, combined with local supervision, constrains the shared parameter space and minimizes unstable behavior in unobserved regions. This theoretical perspective explains how short-sequence training produces stable long-context capabilities. We evaluate EndPrompt (ET) on LLaMA-family models [ 31 , 15 ] , extending the context window from 8K to 64K. RULER [ 19 ] and LongBench [ 2 ] experiments show our method achieves competitive or superior performance compared to baselines such as LCEG [ 25 ] , LongLoRA [ 9 ] , and full-length fine-tuning. Specifically, our method achieves an average RULER score of 76.03, outperforming LCEG (72.24), LongLoRA (72.95), and full-length fine-tuning (69.23). On LongBench, our method secures the highest average score and demonstrates strong performance across tasks including question answering, summarization, few-shot learning, and code completion. Ablation studies validate the effects of end-prompt design, base model choice, extension length, and training-token quantity. These results confirm that reliable long-context adaptation emerges from structured sparse positional supervision without full-length training sequences. The main contributions of this work are summarized as follows: • We propose EndPrompt (ET) , an efficient context-extension method utilizing short training sequences, positional index manipulation, and an appended end prompt to simulate long-range positional supervision. • We demonstrate that preserving the original context as an undivided segment maintains semantic continuity, while the end prompt provides a stable terminal anchor to create long-distance relationships without disrupting the signal for next-token prediction. • We analyze the method through Rotary Position Embedding and position interpolation, explaining how smooth positional variation and shared parameters of the Transformer support generalization over unobserved intermediate distances. • We demonstrate strong empirical performance on RULER and LongBench, where our approach