Stacked from One: Multi-Scale Self-Injection for Context Window Extension

Wei Han, Pan Zhou, Shuicheng Yan · 2026

Read Stacked from One: Multi-Scale Self-Injection for Context Window Extension by Wei Han, Pan Zhou, Shuicheng Yan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04759.txt.

Introduction

Han and colleagues propose SharedLLM, a framework extending LLM context windows via multi-grained compression. Continual pre-training on long-context data is straightforward but costly. SharedLLM stacks two short-context LLMs from the same underlying layers: a lower compressor and an upper reader. The lower model squeezes long inputs into compact multi-grained codes for context-aware processing above. Information transfer happens only at the lowest layers, skipping lengthy forward passes and cross-attention. A tree-based data structure supports fast encoding and query-aware lookup of contextual information.

Catalog record

CREATORWei Han, Pan Zhou, Shuicheng Yan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57398
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04759

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Stacked from One: Multi-Scale Self-Injection for Context Window Extension Wei Han, Pan Zhou, Shuicheng Yan Abstract The limited context window of contemporary large language models (LLMs) remains a primary bottleneck for their broader application across diverse domains. Although continual pre-training on long-context data offers a straightforward solution, it incurs prohibitive data acquisition and computational costs. To address this challenge, we propose SharedLLM , a novel framework based on multi-grained context compression and query-aware information acquisition. SharedLLM comprises two stacked short-context LLMs: a lower model serving as a compressor and an upper model acting as a decoder. The lower model compresses long inputs into compact, multi-grained representations, which are then forwarded to the upper model for context-aware processing. To maximize efficiency, this information transfer occurs exclusively at the lowest layers, bypassing lengthy forward passes and redundant cross-attention operations. This entire process, wherein the upper and lower models are derived from the same underlying LLM layers, is termed self-injection . To support this architecture, a specialized tree-based data structure enables the efficient encoding and query-aware retrieval of contextual information. Despite being trained on sequences of only 8K tokens, SharedLLM effectively generalizes to inputs exceeding 128K tokens. Across a comprehensive suite of long-context modeling and understanding benchmarks, SharedLLM achieves performance superior or comparable to strong baselines, striking an optimal balance between efficiency and accuracy. Furthermore, these design choices allow SharedLLM to substantially reduce the memory footprint and yield notable inference speedups ( 2 × 2\times over streaming and 3 × 3\times over encoder-decoder architectures). The core implementation code, along with training and evaluation details, is open-sourced at https://github.com/Clement25/SharedLLM . More detailes are provided in the appendix and supplementary materials. 1 Introduction Since the release of GPT-3 ( Brown 2020 ) , the rapid advancement of large language models (LLMs) ( Chowdhery et al. 2022 ; Achiam et al. 2023 ; Touvron et al. 2023a ; Touvron et al. 2023b ; Dubey et al. 2024 ; Ma et al. 2024 ; Guo et al. 2025 ) has revolutionized the NLP research community and transformed various workflows. Pretrained on trillions of tokens, LLMs exhibit remarkable abilities, such as completing unfinished text or code and following human instructions to perform designated tasks after simple supervised fine-tuning ( Wei et al. 2021 ; Chung et al. 2024 ; Wang et al. 2025 ) . Despite their impressive capabilities, several factors limit their broader application. One major constraint is the context window size ( Hsieh et al. 2024 ; Liu et al. 2025 ) , which refers to the maximum number of tokens on which an LLM can behave normally. When the input text exceeds this limit, LLMs may suffer from severe performance degradation or hallucination during inference. Many researchers have attempted to extend the context window of LLMs with minimal training costs ( Peng et al. 2023 ; Together 2023 ; Xiong et al. 2024 ) . One common approach involves post-pretraining LLMs on long-context corpora with substantial computational resources ( TogetherAI 2023 ; Xiong et al. 2024 ; Ma et al. 2024 ) . Advanced positional encoding methods, which usually extend RoPE to rescale attention scores in a more principled manner, are integrated to minimize the size of training corpus ( Chen et al. 2023 ; Peng et al. 2023 ) . Although they achieve extrapolation— “train short, test long” , the efficiency is relatively low. For example, to reach the context length of 128K tokens, using YaRN Peng et al. 2023 , one has to pretrain an LLM on 64K tokens. Prompt compression ( Ge et al. 2023 ; Gao et al. 2025 ) accelerates the inference process by replacing long prompts with LLM-generated semantic tokens, but fails to extend the context window of LLMs or only applies on limited scenarios. Other approaches upgrade the conventional transformer architectures to enable streaming processing of long context ( Xiao et al. 2024b ; Yen et al. 2024 ; Zhang et al. 2025 ) , which maintain a sliding window of constant-sized memory. Although these designs significantly alleviate the memory-bound issue of matrix multiplication, their specialized attention patterns may cause incompatibility with high-performance attention implementations (e.g., FlashAttention ( Dao et al. 2022 ; Dao 2023 ) ), potentially leading to slower inference speeds. To strike a balance between efficiency and performance, we propose SharedLLM , a lightweight architecture which consists of one upper model and one lower model . The lower model compresses text chunks into multi-grained representations, while the upper model integrates the encoded information and generates the final output. This multi-grained setting helps LLM focus on task-related fine-grained information while regulating other auxiliary coarse-grained information to a secondary role. Both models are initialized from the same off-the-shelf checkpoint of a short-context LLM, either in full or in part. Since there is no disparity between the hidden spaces of the two models, SharedLLM can be effectively fine-tuned without requiring extensive warmup stages. This paper makes the following major contributions: • We propose SharedLLM , a hierarchical architecture for efficient LLM context window extension. It consists of two models which work collaboratively through shared key-value mechanism with minimal tunable parameters. • We design a tree-like structure, called context tree , which can express long unstructured context in a coarse-to-fine format. To facilitate this process, we introduces a dynamic context tree construction and search algorithm. Given a context and an query, it can efficiently transform the context into the hierarchical representation and collect relevant information from that tree. • We conduct a comprehensive experimental study to demonstrate the effectiveness of SharedLLM . On the settings of both post-pretraining and supervised fine-tuning, SharedLLM shows impressive extrapolation property and yields stronger performance than baseline models with superior memory and time efficiency. 2 Method Figure 1: Overview of SharedLLM . The architecture resembles general encoder-decoder architecture like T5 ( Raffel et al. 2020 ) , but the interaction occurs at the first M layers between lower and upper model through shared key-values which are encoded and compressed from the text chunk into a sequence of trees (top-left). In this section, we first introduce the overall architecture of our proposed SharedLLM in Sec. 2.1 , and then elaborate on its two main components, the lower model and upper model in Sec. 2.2 and 2.3 . 2.1 Overview As illustrated in Figure 1 , SharedLLM adopts a hierarchical architecture. The lower model , or the “compressor”, breaks down the long input context X_{C} into smaller chunks that are then encoded within limited GPU memory. It then uses the same LLM to compress each context chunk into compact and structured coarse-to-fine representations in parallel. The upper model , or the “decoder”, takes the rear part of the input text (the running context, such as questions) as input. It then integrates the compressed information from the lower model, and finally generates predictions of successive tokens in an auto-regressive manner. The lower and upper models are interconnected via the sharing of key-value (KV) states, which are further integrated at the cross-attention modules in the upper model. To facilitate efficient information gathering and integration, the contextual information processed by the lower model is organized as a binary tree, referred to as the context tree , which stores multi-grained

Read the complete text