Large Language Model as Token Compressor and Decompressor

Wenbing LiZikai SongJielei ZhangTianhao ZhaoJunkai LinHuipeng GuoYiran Wang, Wei Yang · 2026

Read Large Language Model as Token Compressor and Decompressor by Wenbing LiZikai SongJielei ZhangTianhao ZhaoJunkai LinHuipeng GuoYiran Wang, Wei Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25340.txt.

Introduction

Li and co-authors turn a stock language model into a text compressor with small add-on adapters. Their self-expressive setup squeezes long texts into short chains of learned codes, called Z-tokens. Dense passages earn more codes, while padded parts get few, under a budget-aware penalty. The same model then expands the codes back into text or task answers. On news, encyclopedia, and question sets, quality holds while context length, memory, and delay all shrink. Work can even proceed inside the compressed space.

Catalog record

CREATORWenbing LiZikai SongJielei ZhangTianhao ZhaoJunkai LinHuipeng GuoYiran Wang, Wei Yang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39227
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25340

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Large Language Model as Token Compressor and Decompressor Wenbing LiZikai SongJielei ZhangTianhao ZhaoJunkai LinHuipeng GuoYiran Wang, Wei Yang Abstract In this paper, we establish the novel insight that an off-the-shelf LLM can function as an excellent token compressor and decompressor. To demonstrate, we design a self-expressive autoencoding learning framework fine-tunes a pretrained LLM to translate long texts into a compact internal language of discrete, variable-length latent codes, termed Z-tokens, and to reconstruct the original text exactly from them. The resulting representation is content-adaptive: semantically dense segments receive more Z-tokens, while redundant or predictable regions are aggressively compressed, via lightweight LoRA-based adapter heads. Empirically, our method achieves up to 18 × \times token reduction on Wikipedia, CNN/DailyMail, HotpotQA, and Qulac-style long-query datasets, while preserving reconstruction fidelity and downstream performance. This simple yet effective design supports applications including prompt compression and autoregressive generation directly in the Z-token space, offering a potential pathway toward token-efficient long-context reasoning. 0 0 footnotetext: ∗ Corresponding author 1 Introduction Large language models (LLMs) have achieved remarkable success in various fields Brown et al. 2020 ; Touvron et al. 2023 . However, their ability to handle long contexts remains fundamentally limited by the quadratic complexity of the attention mechanism Vaswani et al. 2023 . As sequence lengths grow to tens or even hundreds of thousands of tokens, the computational and memory overhead of the attention mechanism makes large-scale inference, long document understanding, and multi-hop retrieval prohibitively expensive. To mitigate this, significant research has focused on token compression, which aims to distill lengthy sequences into shorter, information-dense representations Ge et al. 2024 ; Snell et al. 2022 ; Chevalier et al. 2023 ; Mu et al. 2024 . Existing approaches can be broadly categorized into two categories. Heuristic token merging methods, such as similarity-based merging Bolya et al. 2023 ; Zhong et al. 2025 , attention-based pruning Kim et al. 2022 , and saliency-guided aggregation Li et al. 2023 , attempt to remove or merge redundant tokens based on local similarity or attention importance scores. While these techniques are effective in shortening sequence lengths, tokens that appear redundant due to similarity may carry discourse or pragmatic meaning that is difficult to recover. The second category includes learning-based or autoencoding methods such as AutoCompressor Chevalier et al. 2023 , ICAE Ge et al. 2024 , or Gist Tokens Mu et al. 2024 . AutoCompressor and ICAE generate fixed-length compressed context vectors that can be re-expanded for downstream tasks by training auxiliary networks or soft embedding modules. Gist Tokens force the model to compress cues into fixed-length lexical units through an attention masking mechanism. These methods often treat compression as an external, auxiliary task, decoupled from the LLM’s primary generative reasoning process. The resulting continuous embeddings or latent codes are non-lexical and lack transparency, making them difficult to interpret or control. Moreover, their reliance on fixed compression ratios prevents adaptation to varying information density and typically forgoes faithful reconstruction of the original input, limiting their applicability in scenarios that demand verifiability or fine-grained access to the source content. In this work, we depart from prevailing heuristic and learned-embedding approaches by introducing a novel insight: leveraging the semantic knowledge of the LLM to serve as both a token compressor and decompressor. Our method is predicated on the fundamental insight that LLMs, trained on vast corpora, are not merely statistical models but powerful engines for semantic abstraction and conceptual distillation Wei et al. 2022 ; Bommasani et al. 2022 ; Delétang et al. 2024 . We observe that natural language is inherently compressive; it exhibits a strong tendency to encode complex ideas into compact lexical units. Idioms such as “double-edged sword” encapsulate intricate trade-offs, and metaphors like “tip of the iceberg” summarize extensive underlying complexities. This phenomenon implies that LLMs itself is capable of efficient compression. Methodologically, we propose an autoencoding training paradigm, where an input sequence X is mapped into a compressed sequence of discrete Z-tokens, Z , by a LLM compressor, and subsequently reconstructed or continued by a LLM decompressor. The training objective can be viewed as a form of bilingual translation, where the model learns to ’encode’ natural language into its own abstract, internal language and ’decode’ it back. The compressed latent \{Z\} serves as a semantic scaffold, forcing the model to perform its core reasoning—such as logic, inference, and narrative planning—on the compact symbolic representation before projecting it back to the surface token level. This mechanism ensures that compression is an act of semantic distillation rather than rote memorization; the model must first understand the input’s meaning to produce an effective Z-token sequence that supports subsequent generation. Our design is more flexible, capable of dynamically lengthening compressed text based on semantic density, and seamlessly integrating with sliding windows to support the processing of longer texts. In terms of task design, we support not only text reconstruction and generation but also various downstream tasks such as summarization and long-text QA. We discovered and explained the polysemy of Z-tokens, an interesting phenomenon, and experimentally demonstrated that it is controllable rather than chaotic, providing a novel perspective on text compression. We conducted extensive experiments on six large-scale datasets—Wikipedia, CNN/DailyMail, HotpotQA, NarrativeQA, QASPER and QuALITY—for tasks such as text reconstruction, continuation writing, summarization, and long-text question answering. The results show that our method achieves state-of-the-art performance on all tasks, improves inference speed by 4.6 \times , and significantly reduces memory usage, fully validating the effectiveness and efficiency of the method. 2 Related Work Figure 1 : Our framework consists of three components: a compressor, an inference module, and a decompressor. It supports two usage paradigms. In the first, we directly feed the compressed representation to the decompressor to perform the downstream task. In the second, we first run the inference module in the Z-token space and then invoke the decompressor to recover task outputs at the token level. Context Compression refers to any mechanism that explicitly aims to reduce the number of tokens participating in subsequent computation. Some approaches follow heuristic token reduction strategies, removing or merging less informative tokens based on saliency or similarity. Token Merging Bolya et al. 2023 ; Liu et al. 2025 ; Zhong et al. 2025 aggregates similar tokens , while Token Pruning Kim et al. 2022 ; Zhong et al. 2025 ; Rao et al. 2021 ; Kwek and Yin 2025 ; Song et al. 2024 eliminates low-saliency tokens during inference. Beyond rule-based methods, learning-based semantic compression encodes contexts into compact representations that retain high-level meaning. ICAE Ge et al. 2024 and AutoCompress Chevalier et al. 2023 represent long sequences as latent vectors for efficient reconstruction or reasoning; other recent works similarly investigate learned context compression Snell et al. 2022 ; Zhang et al. 2024 . A related branch, prompt compression , targets input instructions rather than the full context. Methods such as 500xCompressor Li et al. 2024 , Gist Tokens Mu et al. 2024 and LongLLMLingua Jiang et al. 2024 summarize long prompts into shorter, reusable representations. Our method is conceptually aligned with learning-based semantic compression but differs in execution: rather than encoding context externally, we integrate compression into the LLM’s reasoning process. This yields discrete internal symbols that form a self-expressive “semantic language,” achieving adaptive,

Read the complete text