Latent Context Compilation: Distilling Long Context into Compact Portable Memory
Zeju Li, Yizhou Zhou, Qiang Xu · 2026
Read Latent Context Compilation: Distilling Long Context into Compact Portable Memory by Zeju Li, Yizhou Zhou, Qiang Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-21221.txt.
Introduction
Long-context field use stalls between spread-out squeezing, which carries over poorly out of spread, and test-time training, which costs fake data and state-keeping weights. Zeju Li, Yizhou Zhou, and Qiang Xu shift the frame in this 2026 arXiv paper from tuning to building. A throwaway LoRA module acts as builder, boiling long contexts down into compact buffer tokens: state-free, portable memory for frozen base models.
A self-matched tuning plan removes the need for fake question-answer pairs, steadying rebuild with context-blind random queries so squeezed tokens stay within the model's prompt-obeying space. Tests with a popular eight-billion-dial model keep fine detail and thought where prior methods falter, even at 16-fold squeezing. Memory density is unlinked from model dials.
Catalog record
| CREATOR | Zeju Li, Yizhou Zhou, Qiang Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-21221
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Latent Context Compilation: Distilling Long Context into Compact Portable Memory Zeju Li, Yizhou Zhou, Qiang Xu Abstract Efficient long-context LLM deployment is stalled by a dichotomy between amortized compression, which struggles with out-of-distribution generalization, and Test-Time Training, which incurs prohibitive synthetic data costs and requires modifying model weights, creating stateful parameters that complicate concurrent serving. We propose Latent Context Compilation , a framework that fundamentally shifts context processing from adaptation to compilation. By utilizing a disposable LoRA module as a compiler, we distill long contexts into compact buffer tokens —stateless, portable memory artifacts that are plug-and-play compatible with frozen base models. Crucially, we introduce a self-aligned optimization strategy that eliminates the need for synthetic context-relevant QA pairs. By regularizing context reconstruction task with context-agnostic random queries, we force compressed tokens to reside within the model’s existing instruction-following manifold. Experiments with Llama-3.1-8B demonstrate that Latent Context Compilation preserves fine-grained details and reasoning capabilities where prior methods falter, effectively decoupling memory density from model parameters even at a 16 × \times compression ratio. † † affiliationnotice: †Corresponding Author 1 Introduction While LLMs now support million-token windows, the quadratic cost of attention and prohibitive KV cache footprints create a “Context Bottleneck” that renders scalable deployment cost-prohibitive. Consequently, High-Fidelity Context Compression has become imperative to reduce input length without compromising semantic integrity. Current approaches generally fall into two paradigms, each facing a fundamental trade-off. The first, Amortized Compression (or General Context Compression), learns a fixed policy to compress inputs. Methods like LLMLingua ( 22 ) , 500xCompressor ( 15 ) , and ICAE ( 5 ) are efficient but suffer from a generalization gap: a compressor trained on a pre-defined corpus inevitably struggles to capture the fine-grained nuances of out-of-distribution (OOD) contexts at test time. They often sacrifice detail for brevity, making them unsuitable for precision-critical tasks. To bridge this gap, recent research has pivoted towards Instance-Specific Optimization, leveraging the emerging paradigm of “Test-Time Compute” to adapt models dynamically. This landscape is generally divided into two streams, yet neither fully resolves the efficiency-deployment dilemma. • Test-Time Adaptation (TTA) ( 8 ; 21 ) , optimizes the model to align with the test data distribution. While effective for mitigating domain shifts, TTA primarily targets prediction robustness rather than dimensionality reduction; the raw context is still required as input, leaving the prohibitive computational overhead unresolved. • Test-Time Training (TTT) ( 28 ; 1 ; 19 ) , attempts to address this by “memorizing” specific context content directly into model parameters. However, this introduces significant architectural friction . By encoding memory into weights, these methods transform the LLM from a stateless engine into a stateful entity. While these weights are technically swappable, managing concurrent requests with different parameter states precludes standard KV-caching optimizations and complicates high-throughput serving. Furthermore, this parameter coupling risks degrading general reasoning capabilities due to overfitting on the specific context. Table 1 : Comparison of our Latent Context Compilation with other methods on training stage and inference stage. Method Hierarchy Training Stage Inference Stage Pretraining Context-relevant Queries Optimization Target Inference Dependency Frozen Parameter General Compression LLMLingua2 ( 22 ) ✓ ✗ Global Encoder \theta_{enc} (Fixed) Memory Slots / Soft Prompts ✓ AutoCompressor ( 2 ) ✓ ✗ GIST ( 18 ) ✓ ✗ Test Time Adaption TLM ( 8 ) ✗ ✓ Model Weights \Delta\theta (Distribution) LoRA Weights + Raw Context ✗ SIFT ( 9 ) ✗ ✓ TTT-NN ( 7 ) ✗ ✓ Test Time Training Temp-LoRA ( 28 ) ✗ ✗ Model Weights \Delta\theta (Instance-Specific) LoRA Weights ✗ StreamAdapter ( 19 ) ✗ ✓ InfiniteICL ( 1 ) ✗ ✓ Latent Context Compilation ✗ ✗ Buffer Tokens Buffer Tokens (KV Cache Only) ✓ In this paper, we propose Latent Context Compilation , a novel framework that fundamentally shifts context processing from adaptation to compilation . Synthesizing the instance-specificity of TTT with the deployment efficiency of standard caching, we leverage a disposable LoRA module as a compiler to distill the context into a compact set of Buffer Tokens (Input KV cache). Crucially, this results in Data Portability rather than weight portability: the LoRA is discarded after optimization, leaving a portable memory artifact that can be plugged into the frozen base model as standard input. To ensure this compilation is robust without ground-truth supervision, we introduce a self-aligned optimization strategy: we optimize for Context Reconstruction Task to ensure high fidelity, while simultaneously regularizing the model with context-agnostic random queries to maintain its general instruction-following manifold. A detailed comparison of methodological differences is presented in Table 1 , highlighting how Latent Context Compilation overcomes the stateful limitations of TTT and the generalization gap of Amortized Compression. To validate our approach, we employ Llama-3.1-8B-Instruct ( 6 ) across a diverse suite of benchmarks. On context-specific QA and summarization tasks, Latent Context Compilation significantly outperforms baselines, confirming that our buffer tokens successfully encode both the fine-grained details and global semantics required for high-fidelity retrieval. Crucially, on general reasoning benchmarks, we demonstrate that the model maintains performance parity with the original base model. This proves that our context self-aligned training successfully prevents the catastrophic forgetting common in prior distillation works, preserving the model’s fundamental capabilities while compressing memory. Our contributions are summarized as follows: • We introduce Latent Context Compilation , a framework that uses a disposable LoRA to compile contexts into portable Buffer Tokens. This decouples memory from model weights, overcoming the stateful serving bottlenecks of Test-Time Training. • We propose a self-aligned optimization strategy that combines context reconstruction task with agnostic queries regularization. This prevents manifold collapse and ensures high-fidelity compression without the cost or bias of synthetic data. • Experiments demonstrate that Latent Context Compilation achieves a 16 \times - 32 \times compression ratio with significantly higher retention than extractive and weight-based baselines, while fully preserving general reasoning capabilities. 2 Related Work 2.1 General Context Compression Current strategies for long-context efficiency primarily rely on learning a general-purpose compression policy over large-scale datasets, generally falling into extractive pruning or amortized soft compression categories. Extractive pruning strategies, such as LLMLingua ( 10 ) , LLMLingua-2 ( 22 ) , KV cache compression ( 17 ; 4 ) and Selective Context ( 13 ) , operate by employing budget controllers or distilled token classifiers to explicitly identify and remove tokens deemed redundant based on information-theoretic metrics. However, physically severing tokens from the sequence risks disrupting long-range dependencies and diminishing context integrity ( 14 ; 32 ) , particularly in tasks requiring fine-grained reasoning. Alternatively, amortized soft compression approaches focus on condensing text into compact latent representations or memory slots to preserve semantic information. This paradigm encompasses autoencoder-based models like ICAE ( 5 ) and AutoCompressors ( 2 ) , which leverage massive pre-training on autoencoding objectives to encode long contexts into summary vectors. Other works explore soft prompt learning, such as Gist Tokens ( 18 ; 3 ) and contrastive conditioning ( 29 ) , or introduce specialized compression modules, including Activation Beacon ( 31 ) , retrieval-augmented compressors like RECOMP ( 30 ) , and lightweight adapters for streaming contexts ( 11 ; 16 ) . While these amortized methods offer efficient inference, they fundamentally suffer from the generalization gap : they rely on a fixed compression