Compressing Sequences in the Latent Embedding Space: K K -Token Merging for Large Language Models
Zihao Xu, John Harvill, Ziwei Fan, Yizhou Sun, Hao Ding, Hao Wang · 2026
Read Compressing Sequences in the Latent Embedding Space: K K -Token Merging for Large Language Models by Zihao Xu, John Harvill, Ziwei Fan, Yizhou Sun, Hao Ding, Hao Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15153.txt.
Introduction
Xu and colleagues shrink long prompts inside the model's embedding space rather than in raw token space. Their K-Token Merging folds each block of K token embeddings into one embedding through a light encoder, then runs the shortened sequence through a LoRA-tuned model. Generation stays in the original vocabulary, so output form is unchanged. Tests on reasoning, sentiment, and code-editing tasks reach the performance-compression frontier, cutting input length by up to 75 percent with little quality loss.
Catalog record
| CREATOR | Zihao Xu, John Harvill, Ziwei Fan, Yizhou Sun, Hao Ding, Hao Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29199 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15153
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Compressing Sequences in the Latent Embedding Space: K K -Token Merging for Large Language Models Zihao Xu, John Harvill, Ziwei Fan, Yizhou Sun, Hao Ding, Hao Wang Abstract Large Language Models (LLMs) incur significant computational and memory costs when processing long prompts, as full self-attention scales quadratically with input length. Token compression aims to address this challenge by reducing the number of tokens representing inputs. However, existing prompt-compression approaches primarily operate in token space and overlook inefficiencies in the latent embedding space . In this paper, we propose K K -Token Merging, a latent-space compression framework that merges each contiguous block of K K token embeddings into a single embedding via a lightweight encoder. The compressed sequence is processed by a LoRA-adapted LLM, while generation remains in the original vocabulary. Experiments on structural reasoning (Textualized Tree), sentiment classification (Amazon Reviews), and code editing (CommitPackFT) show that K K -Token Merging lies on the Pareto frontier of performance vs. compression, achieving up to 75% input length reduction with minimal performance degradation. Code is available at https://github.com/shsjxzh/K-Token-Merging . 1 Introduction Figure 1: Our K -Token Merging method ( K=4 ) achieves a 75\% reduction in input length with only a 1.59\% drop in accuracy on the Textualized Tree benchmark, demonstrating that it exploits redundancy in the latent embedding space while preserving high performance. See the “Experiments” section for details. We have witnessed tremendous progress in Large Language Models (LLMs) ( 33 ) . Since the advent of ChatGPT ( 2 ) , LLMs have become deeply integrated into many aspects of daily life – from AI-enhanced customer support ( 29 ; 27 ) and game NPCs ( 7 ; 5 ) to agentic systems capable of data analysis ( 30 ) and complex software generation ( 15 ; 14 ) . These increasingly sophisticated applications demand ever-longer input contexts, which sharply escalate both computational and memory costs: with softmax attention ( 31 ) , the memory and compute requirements of LLMs grow quadratically with input length. A natural direction to mitigate this challenge is token compression ( 19 ) . Token compression aims to reduce the input length by representing prompts with fewer tokens. Existing approaches largely fall into two categories. The first category, hard prompt compression ( 18 ; 25 ; 13 ; 20 ) , decreases token count by dropping or summarizing tokens. While effective for tasks where key information is sparse, these methods often fail on information-dense tasks such as document revision, mathematical reasoning, symbolic computation, or code translation–settings in which every token may carry essential information. Figure 2: Model Structure for K -Token Merging Model (Case K=2 ). Left: During the prefill stage, the encoder f takes each K consecutive input tokens and produces a single compressed token embedding. Here, T_{i} denotes the original input tokens and C_{i} denotes the resulting compressed tokens. Right: During the generation stage, the LLM outputs original (uncompressed) tokens. Each newly generated token is appended to the mixed compressed/uncompressed prefix, after which standard auto-regressive generation continues. Here, G_{i} denotes the generated uncompressed tokens. The second category, soft prompt compression ( 23 ; 10 ; 8 ) , draws inspiration from soft prompts ( 17 ) and learns new tokens or adapts model parameters to produce more compact representations. These methods have gained increasing attention because, empirically, they preserve more information than hard methods while still offering substantial compression ( 10 ; 8 ) . However, existing methods focus almost exclusively on reducing the number of tokens , overlooking a major source of redundancy: the embedding space itself . Consider the Qwen 2.5 model ( 26 ) , which has a vocabulary size of 151{,}936 . Each token consumes approximately 896\times 32=28{,}672 bits, whereas the theoretical minimum needed to identify one token from its vocabulary is only \log_{2}(151{,}936)\approx 18 bits. For a K -gram, the minimum grows to merely \log_{2}(151{,}936^{K})\approx 18K bits. This enormous gap highlights significant inefficiency in current embeddings and indicates substantial room for improving input representation. In this paper, we propose K -Token Merging , a new soft prompt compression method that directly targets redundancy in the latent embedding space . Our key idea is to use a learned encoder to compress a sequence of K consecutive tokens ( K -grams) into a single embedding. In other words, we represent multiple tokens with one embedding on the input side only . We then finetune the LLM with LoRA to adapt to these compressed embeddings, enabling strong downstream performance. As illustrated in Figure 1 , our approach achieves high compression ratios with minimal performance degradation (see the “Experiments” section for details). Importantly, our method is fundamentally different from expanding the vocabulary with K -grams. Vocabulary expansion requires representing multi-token sequences as single tokens on both the input and output sides . This leads to an explosion of low-frequency “tail” tokens, which severely complicates optimization and becomes quickly infeasible for large K . For example, the number of unique 4-grams can exceed the size of the original vocabulary by 29\times ( 32 ) , resulting in prohibitive storage overhead. Furthermore, since each K -gram is assigned a fixed embedding, such approaches cannot generalize to unseen K -grams during inference, sharply limiting their applicability. In contrast, our encoder-based approach avoids tail-token explosion, supports large K values without increasing vocabulary size, and generalizes naturally to unseen token combinations. Our main contributions can be summarized as follows: Figure 3: Datasets & Tasks. Left: Textualized Tree. Given a textualized indentation tree, the LLM determines whether two nodes have a parent - child relationship. Middle: Amazon Reviews. The LLM performs sentiment classification to judge whether a product review is positive or negative. Right: CommitPackFT. Given code and an update instruction, the LLM needs to output the modified code that follows the instruction. • We introduce K -Token Merging , a novel framework for latent-space token sequence compression. • We present a training recipe for adapting LLMs to such compressed inputs. • We show experimentally that our approach achieves up to 75% token compression with minimal performance degradation across diverse tasks. In scenarios where output length is much shorter than input length, this corresponds to an estimated 94% 1 1 1 With K=4, input length is reduced by 75%. Prefill attention cost scales as O(n²), so compressed prefill requires only 0.25^{2}=6.25\% of the original FLOPs. When output length M\ll input length KN , total FLOPs \approx 6.25\% of uncompressed, yielding \sim 94\% reduction. reduction in computation . 2 Related Work Token compression ( 19 ) has been extensively studied since the introduction of the Transformer architecture ( 31 ) . Existing approaches can be grouped into hard prompt compression and soft prompt compression . Hard prompt compression. Hard compression methods ( 18 ; 25 ; 6 ; 13 ; 16 ; 28 ; 21 ; 20 ) reduce input length by dropping or summarizing tokens. For example, SelectiveContext ( 18 ) removes tokens deemed uninformative based on self-information, and LLMLingua2 ( 25 ) trains a classifier to identify redundant tokens. However, these methods often suffer significant performance degradation on information-dense tasks, as even a small number of omitted tokens may contain essential context. In contrast, our method never removes