Sentence-Anchored Gist Compression for Long-Context LLMs

Dmitrii Tarasov, Elizaveta Goncharova, Kuznetsov Andrey · 2025

Read Sentence-Anchored Gist Compression for Long-Context LLMs by Dmitrii Tarasov, Elizaveta Goncharova, Kuznetsov Andrey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08128.txt.

Introduction

Dmitrii Tarasov, Elizaveta Goncharova, and Andrey Kuznetsov study learned tokens that compress the context a large language model must carry. Long inputs strain memory and computation, so shrinking context without losing task skill is valuable. Their sentence anchored gist approach fine tunes pretrained models to pack context into far fewer tokens.

Compression factors from two to eight times are reported without major performance loss on short and long context benchmarks. On a three billion parameter LLaMA model, the method matches rival compression techniques while reaching higher ratios. The result suggests trained models can summarize their own context internally rather than relying on outside trimming tools.

Catalog record

CREATORDmitrii Tarasov, Elizaveta Goncharova, Kuznetsov Andrey
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-08128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sentence-Anchored Gist Compression for Long-Context LLMs Dmitrii Tarasov, Elizaveta Goncharova, Kuznetsov Andrey Abstract This work investigates context compression for Large Language Models (LLMs) using learned compression tokens to reduce the memory and computational demands of processing long sequences. We demonstrate that pre-trained LLMs can be fine-tuned to compress their context by factors of 2x to 8x without significant performance degradation, as evaluated on both short-context and long-context benchmarks. Furthermore, in experiments on a 3 - billion - parameter LLaMA model, our method achieves results on par with alternative compression techniques while attaining higher compression ratios. 1 Introduction The growing demand for processing long-context inputs in Large Language Models (LLMs) is often bottlenecked by memory and computational requirements of the self-attention mechanism. In a recent paper 14 showed that LLMs possess a significant capacity for compression, with an 8B model capable of condensing up to 1568 tokens into a single vector in a prompt-tuning fashion (smaller models achieve lower rates, e.g., 128 tokens per vector). We argue that this inherent ability can be extended beyond prompt-tuning. Instead of a single compression token, we propose learning a set of compression tokens that pool the most salient information from distinct text segments. These tokens then serve as the conditioning context for subsequent segments during language modeling, effectively creating an information bottleneck, where each segment primarily attends to the compressed representation of the previous one. This architecture can be implemented with minimal modifications to the standard transformer: by extending the model’s vocabulary to include the new compression tokens and adjusting the attention mask to enforce the segment-conditioning pattern. This stands in contrast to methods like 28 proposed, which, while powerful, originally required Backpropagation Through Time (BPTT) — though a mask-based implementation is also feasible. A key advantage of our setup is that the model learns to compress the context end-to-end using only the standard Language Modeling (LM) objective, eliminating the need for an auxiliary reconstruction loss as proposed by 7 to improve uniform gist-based compression. Furthermore, while preceding works typically employ a uniform distribution of compression tokens ( 28 ; 7 ) , we introduce a rule-based strategy for token insertion at the end of each sentence. This data-dependent positioning aims to align compression boundaries with natural semantic units, thereby facilitating more meaningful and coherent information aggregation. In summary, our contributions are as follows: • We propose a novel, rule-based strategy for positioning compression tokens to enable data-dependent and semantically-aware context compression. • We implement the proposed method via a simple attention mask modification, which enables efficient parallel processing during both training and prefilling, enhancing scalability. • We extensively evaluate the proposed framework and demonstrate that it maintains strong performance with no significant degradation on both short- and long-context benchmarks. • We achieve high KV-cache compression rates, ranging from 2x to 8x, across various evaluation benchmarks. 2 Related Work Sparse Attention. Sparse attention methods aim to reduce computational costs by limiting the number of tokens to which each token can attend. In Native Sparse Attention 25 proposed a hardware-aligned hierarchical sparse attention that combines coarse-grained token compression with fine-grained token selection. The Forgetting Transformer ( 16 ) learns to limit local attention spans via precomputed forgetting scores. Recurrence. Recurrent methods compress context into a fixed state for long-range dependency modeling. The Recurrent Memory Transformer ( 2 ) learns memory embeddings in an RNN-like fashion, significantly reducing memory requirements. AutoCompressors ( 5 ) extend this idea by learning to generate compression tokens that are used as soft prompts. More recently, the In-context AutoEncoder ( 9 ) uses an autoregressive encoder to compress context into learned tokens for a decoder, achieving up to 4x compression without BPTT. While effective, many recurrent approaches require Backpropagation Through Time, which can slow training and hinder parallelization. KV-Cache Compression. These methods reduce the memory footprint of the Key-Value cache during generation. 3 uses redundancy estimation and importance scoring to retain approximately 10% of the cache. 15 employs clustering of attention patterns for an 8.2x memory reduction. 29 evicts tokens based on attention scores, reducing the cache by up to 5x. 22 merges cache entries via clustering, achieving 3x compression. FastGen ( 8 ) profiles and applies head-specific strategies (e.g., punctuation, locality) to halve the cache budget. StreamingLLM ( 23 ) uses attention sinks and a sliding window ( 1000 tokens) for streaming applications. SepLLM ( 4 ) extends this by caching separator tokens (e.g., punctuation), showing it generalizes StreamingLLM and improves performance, though it requires task-specific window tuning. Gist Token Compression. This line of work introduces learned “gist” or “beacon” tokens to summarize context. The most related to ours is 28 , which processes context in chunks with interleaved beacon tokens to accumulate information. In contrast, our method requires only a single forward pass for the prefill stage and employs data-dependent compression at sentence boundaries. 19 compresses task-specific prompts into gist tokens via a modified attention mask, achieving up to 26x compression. 7 explores different compression rates and token placement strategies, using an autoencoding loss to improve compression quality. 3 Sentence Transformer The proposed Sentence transformer takes inspiration from gist or beacon token compression ( 28 ; 7 ) and data-dependent compression from SepLLM ( 4 ) . 3.1 Learned Gist Tokens We extend the LLM’s embedding vocabulary by adding N_{g} new gist tokens. These tokens are initialized by sampling from a multivariate normal distribution whose parameters ( \mu , \Sigma ) are derived from the existing vocabulary embeddings, following the “mean-resizing” approach by 13 . The language modeling head is resized correspondingly. When the LM head is tied to the input embeddings (as is common in smaller models (e.g., 1–3B parameters)) we resize the shared matrix jointly; otherwise, the two matrices are resized separately. 3.2 Gist Tokens Placement The gist tokens in this work serve as aggregators for some text segments. This work employs a simple, rule-based strategy for positioning compression tokens. Specifically, gist tokens are inserted at the end of each sentence, anchored to standard sentence-ending punctuation marks such as ‘.’, ‘!’, and ‘?’. This approach aims to align compression boundaries with natural semantic units. Processing Example For N_{g}=2 gist tokens: Original: The sun was shining brightly. Birds were singing in the forest. Processed: The sun was shining brightly. < g_{1} >< g_{2} > Birds were singing in the forest. < g_{1} >< g_{2} > 3.3 Attention Mask Within the transformer architecture, the gist tokens for a given sentence are allowed to attend to all tokens within that same sentence, as well as to all gist tokens from preceding sentences. This enables the aggregation of both local sentence-level information and global context from the compressed history. Figure 1(b) illustrates the sentence attention pattern, which exhibits sparsity despite an increased sequence length. This expansion occurs because gist tokens are added to the initial sequence. (a) Causal attention (b) Sentence attention Figure 1: Comparison of attention mechanisms: (a) causal attention, and (b) sentence attention with N_{g}=1 gist token. Gist tokens ( g_{1} ) are inserted at sentence boundaries and pool information from their entire sentence. They are visible to all subsequent tokens, while regular tokens ( t_{i} ) only attend within their sentence. 3.4 Objective The training methodology relies exclusively

Read the complete text