Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation
Zichong Li, Chen Liang, Liliang Ren, Tuo Zhao, Yelong Shen, Weizhu Chen · 2026
Read Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation by Zichong Li, Chen Liang, Liliang Ren, Tuo Zhao, Yelong Shen, Weizhu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14339.txt.
Introduction
Li and five colleagues propose RoPE-Perturbed Self-Distillation, a regularizer for positional robustness in long-context adaptation. Standard adaptation stays brittle: accuracy hinges on evidence placement, with high positional variance at fixed format and difficulty. The method builds alternate views of each sequence by perturbing RoPE indices, moving context parts to new positions, and trains consistent predictions across views by self-distillation. Tests on Llama-3-8B and Qwen-3-4B show steady gains on benchmarks, including RULER-64K and RULER-256K, plus better length extrapolation beyond the training window.
Catalog record
| CREATOR | Zichong Li, Chen Liang, Liliang Ren, Tuo Zhao, Yelong Shen, Weizhu Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44161 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14339
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Shuffle the Context: RoPE-Perturbed Self-Distillation for Long-Context Adaptation Zichong Li, Chen Liang, Liliang Ren, Tuo Zhao, Yelong Shen, Weizhu Chen Abstract Large language models (LLMs) increasingly operate in settings that require reliable long-context understanding, such as retrieval-augmented generation and multi-document reasoning. A common strategy is to fine-tune pretrained short-context models at the target sequence length. However, we find that standard long-context adaptation can remain brittle: model accuracy depends strongly on the absolute placement of relevant evidence, exhibiting high positional variance even when controlling for task format and difficulty. We propose RoPE-Perturbed Self-Distillation , a training regularizer that improves positional robustness. The core idea is to form alternative “views” of the same training sequence by perturbing its RoPE indices—effectively moving parts of the context to different positions—and to train the model to produce consistent predictions across views via self-distillation. This encourages reliance on semantic signals instead of brittle position dependencies. Experiments on long-context adaptation of Llama-3-8B and Qwen-3-4B demonstrate consistent gains on long-context benchmarks, including up to 12.04% improvement on RULER-64K for Llama-3-8B and 2.71% on RULER-256K for Qwen-3-4B after SFT, alongside improved length extrapolation beyond the training context window. † † affiliationnotice: * Work is done during internship at Microsoft. (a) Answer position vs. accuracy. (b) Overview. Figure 1 : (a) Relation between answer position and accuracy on the NIAH multikey-2 task from RULER ( 14 ) with 64k-token sequences. We evaluate the ProLong-64k-base model ( 10 ) as causal LM baseline and model trained with our method. (b) Overview of our proposed objective. 1 Introduction Many real-world applications of large language models (LLMs)—including long-horizon software engineering and reasoning ( 17 ) , multi-document question answering ( 3 ) , and retrieval-augmented generation (RAG) ( 11 ) —critically depend on strong long-context performance. In these settings, relevant evidence may appear anywhere within very long input sequences and may shift substantially across queries due to document ordering, concatenation strategy, and retrieval placement. A common way to improve long-context performance is to fine-tune short-context pretrained LLMs on sequences at the target length ( 36 ; 27 ) , often monitoring progress using synthetic needle-in-a-haystack (NIAH) evaluations ( 14 ; 7 ; 33 ) . Despite the effectiveness of such long-context fine-tuning ( 10 ) , we find that the resulting models can remain brittle with respect to evidence placement. Figure 1(a) illustrates this effect on the RULER multikey-2 NIAH task ( 14 ) : a strong long-context fine-tuned baseline exhibits a pronounced position dependence, with accuracy varying with the answer’s location within the prompt. While this behavior does not preclude good average performance, it is undesirable in long-context pipelines such as RAG, codebase/file concatenation, or multi-source packing, where the same evidence span may appear at widely different absolute positions across inputs. This position sensitivity can be viewed through the lens of rotary position embeddings (RoPE; ( 26 ) ), which most modern LLMs use for positional encoding. RoPE rotates query and key vectors by position-dependent angles, so attention depends on relative offsets. In long-context regimes, these position-dependent phases can make model behavior sensitive to how positional indices are assigned. This perspective motivates explicitly encouraging positional robustness during training. We propose RoPE-Perturbed Self-Distillation , a simple regularizer that creates an alternative “view” of each training sequence by shifting RoPE indices for a contiguous span (Figure 1(b) ). Concretely, given a length- L sequence, we sample a split point s and add an offset y to the RoPE indices of all tokens in the suffix [s,L-1] , while keeping [0,s-1] unchanged. This perturbation effectively increases the RoPE distance between the suffix and the earlier prefix by y , so tokens in [s,L-1] “see” the preceding context as farther away. We train the model with two forward passes: a standard view and a perturbed view (Figure 1(b) ). On the standard view, we optimize the usual causal language modeling (CLM) loss. We then enforce cross-view consistency by minimizing a Kullback–Leibler (KL) divergence that matches the perturbed-view next-token distribution to the standard-view distribution, using stop-gradient on the standard path. In this way, the standard view provides a stable reference under the original index assignment, while the perturbed view is trained to align with it, penalizing prediction differences attributable purely to the RoPE shift and thereby reducing sensitivity to evidence placement. The effect of this perturbation can be understood through RoPE’s multi-frequency structure: higher-frequency components vary rapidly with position (capturing fine-grained phase), while lower-frequency components vary more slowly (capturing coarser structure). The index shift primarily disrupts the position-sensitive high-frequency components for interactions involving the shifted span, while leaving coarse structure relatively stable. Enforcing agreement across the two views therefore encourages the model to rely more on position-robust signals—semantic content and coarse positional structure. Figure 2 : Analysis of attention distance patterns on a 64k-token sequence from Prolong ( 10 ) dataset. For both the baseline model and our model, we inspect attention scores from the final 256 query tokens. We consider attention weights above a threshold ( 10^{-3} ) and measure the signed distance between query and key indices. The figure shows the histogram of attention distances in the 24th layer. Empirically, as shown in Figure 1(a) , our RoPE-perturbed self-distillation substantially reduces the variation of accuracy with answer position on the NIAH task, suggesting reduced positional brittleness. Complementing this, our attention analysis on long sequences (Figure 2 ) shows that models trained with our objective allocate more attention mass to larger distances and exhibit a more uniform distribution of long-range attention. In Section 3 , we present a more comprehensive evaluation on long-context adaptation of Llama-3-8B and Qwen-3-4B using the RULER ( 14 ) and HELMET ( 38 ) benchmarks. Across both model families, our method yields consistent gains on long-context benchmarks, including 12.04% on RULER-64K for Llama-3-8B and 2.71% on RULER-256K for Qwen-3-4B after SFT, with similarly positive trends on HELMET. We further observe improved length extrapolation when evaluating at 2–4 \times the training context length using YaRN-based extension ( 23 ) , supporting robustness beyond the trained window. Finally, these improvements come with comparable short-context performance to the baseline. 2 Method 2.1 Problem setting and notation Let x_{0:L-1}=(x_{0},\dots,x_{L-1}) be a token sequence of length L drawn from a long-context corpus \mathcal{D} . We consider an autoregressive Transformer language model with parameters \theta that uses rotary position embeddings (RoPE; ( 26 ) ) in each self-attention layer. We expose RoPE through an explicit index vector r=(r_{0},\dots,r_{L-1})\in\mathbb{Z}^{L} , where token x_{i} is assigned RoPE index r_{i} . Given (x_{0:L-1},r) , the model defines next-token distributions p_{\theta}(\cdot\mid x_{<i};r),\qquad i=0,\dots,L-1, (1) under a causal mask (so token i attends only to <i ). Standard long-context fine-tuning minimizes the causal language modeling (CLM) loss under the unperturbed RoPE indices: r_{i}=i,\qquad i=0,\dots,L-1, (2) \mathcal{L}_{\text{CLM}}(\theta)=\mathbb{E}_{x_{0:L-1}\sim\mathcal{D}}\Big[-\sumop\displaylimits_{i=0}^{L-1}\log p_{\theta}(x_{i}\mid x_{<i};r)\Big]. (3) Our goal is to adapt LLMs to long contexts while reducing brittle dependence on RoPE indices. We do so by constructing perturbed views of the same token sequence—views that keep content and the causal mask unchanged but alter RoPE indices—and enforcing prediction consistency across views via self-distillation. The framework is essentially agnostic to the specific perturbation: any transformation that changes RoPE indices can serve as a view generator. 2.2 RoPE-perturbed views via skip-based