Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction

Yuchun Zou, Junhong Tong, Jun Li · 2026

Read Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction by Yuchun Zou, Junhong Tong, Jun Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-29000.txt.

Introduction

Yuchun Zou, Junhong Tong, and Jun Li study lossy semantic text compression. An encoder deletes parts of a text, and a language model rebuilds the original from the kept skeleton. They benchmark deletion rules including uniform steps, length and frequency guidance, LP optimization, surprisal entropy, and hybrids. On BBC News data, word-frequency deletion proves a strong baseline, competitive with costlier semantic methods and far faster to encode. Semantic and hybrid methods win clearest at moderate compression, while frequency deletion holds up best at the lowest retention rates.

Catalog record

CREATORYuchun Zou, Junhong Tong, Jun Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78929
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.29000

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Text-Preserving Lossy Text Compression: A Study of Strategic Deletion and LLM Reconstruction Yuchun Zou, Junhong Tong, Jun Li Abstract Traditional lossless text compression preserves every byte, but its gains on natural language are often modest in realistic operating regimes. We study lossy semantic text compression , where the encoder strategically deletes parts of the text and a large language model (LLM) reconstructs the original content from the retained skeleton. We benchmark a progression of deletion strategies, including uniform step deletion, word-length-guided deletion (WordLen), word-frequency-guided deletion (WordFreq), LP-optimized deletion (Opt), entropy-based deletion using GPT-2 surprisal, and hybrid methods that combine frequency and surprisal signals. Evaluation on the BBC News dataset across retention rates r keep ∈ [ 0.1 , 0.9 ] r_{\mathrm{keep}}\in[0.1,0.9] shows three main findings. First, WordFreq is a strong low-cost baseline: despite using only a static frequency lookup, it remains competitive with much more expensive semantic methods while being far faster at the encoder. Second, semantic and hybrid methods provide their clearest gains at mild-to-moderate compression, whereas word-frequency deletion is often more robust at the lowest retention rates. Third, QLoRA fine-tuning yields a strong local decoder that is competitive with Gemini 2.0 Flash and is often strongest in decoder-only comparisons. Additional English and Chinese experiments show that the overall framework transfers across domains, while the best deletion rule remains dataset-dependent. 1 Introduction The information capacity of digital networks and storage systems has grown enormously, yet the volume of generated text data continues to outpace these expansions. Traditional lossless compression algorithms, such as zlib ( 6 ) , bzip2 ( 21 ) , and LZMA ( 16 ) , reduce storage requirements by exploiting statistical redundancies at the byte level. However, because they guarantee exact, bit-for-bit reconstruction of the original sequence, their compression gains on natural language are inherently limited and often remain modest in realistic operating regimes. Motivation. Our primary target setting is context- and bandwidth-limited processing of factual text in LLM-centered pipelines. A document or prior interaction history may need to pass through a narrow text channel before later use: for example, before being inserted into a limited context window, cached as agent memory, relayed between text-based components, or uploaded from a weak sender to a stronger receiver. In such settings, the intermediate representation may still need to remain textual rather than become an opaque latent vector, because text is easy to inspect, edit, search, log, and route through existing interfaces. This makes text-preserving lossy compression a distinct operating regime from unconstrained semantic coding. We do not claim that a textual bottleneck is information-theoretically optimal; rather, we study the practical regime where the compressed artifact must remain text and approximate semantic recovery is acceptable. Yet no systematic benchmark exists for this regime, leaving practitioners without principled guidance on deletion strategy, encoder-side cost, or the viability of lightweight local decoders. We address this gap in this paper. Our framework transmits a compressed skeleton obtained by strategically deleting tokens, and delegates reconstruction to an LLM decoder at the receiver. This directly targets settings such as context-window reduction, agent-memory caching, and weak-sender/strong-receiver text transfer, where the artifact being compressed is meant to remain useful as text before or after reconstruction. Unlike abstractive summarization ( 17 ; 20 ; 13 ) , which rewrites the source, our approach preserves original tokens as anchor points for reconstruction. Unlike prompt compression methods such as LLMLingua ( 10 ) and Selective Context ( 12 ) , which optimize downstream task accuracy for a fixed consumer, our goal is to maximize fidelity to the original text after reconstruction. We therefore study reusable lossy text artifacts for later recovery, not only immediate task performance from a compressed prompt. Downstream-task accuracy is thus a useful future extension rather than the primary objective here. Determining which tokens to delete is critical: naïve uniform deletion destroys linguistic context and maximizes ambiguity. We compare a progression of deletion schemes: uniform step deletion, word-length-guided deletion (WordLen), word-frequency-guided deletion (WordFreq), LP-optimized deletion (Opt), entropy-based deletion via GPT-2 surprisal, and hybrid combinations of frequency and surprisal signals. We evaluate via post-reconstruction BERTScore ( 29 ) on English news across retention rates r_{\mathrm{keep}}\in[0.1,0.9] , using Gemini 2.0 Flash (zero-shot) and QLoRA-fine-tuned Llama-3.2-3B-Instruct ( 3 ) as decoders. Cross-domain experiments on additional English datasets (Wikipedia, Reddit) and an extension to Chinese news are reported in the appendix. We also include a length-constrained LLM summarization baseline to separate reconstruction-oriented deletion from a more conventional lossy rewriting alternative. Contributions. (i) We formulate reconstruction-oriented, text-preserving lossy compression as a benchmark setting with a systematic evaluation protocol (BERTScore as primary metric; ROUGE-L, CER, and NER entity preservation in the appendix) across nine retention rates. (ii) We provide a measurement study over a broad family of deletion strategies, showing that their relative strengths are strongly regime-dependent: semantic and hybrid methods help most at mild-to-moderate compression, while WordFreq remains more robust at the lowest retention rates. (iii) We identify low-resource sender-side compression schemes that are practical when encoder compute, memory, power, or bandwidth are constrained: in particular, WordFreq achieves competitive reconstruction quality with only a static lookup, while Hybrid- \alpha offers a stronger but more expensive semantic alternative. (iv) We show that strategy-aware fine-tuning can make a compact local decoder highly competitive with a stronger zero-shot proprietary decoder under the same reconstruction setting, enabling a local deployment path. (v) Cross-domain experiments on four datasets (see Appendices A – C ) show that several qualitative trends transfer, but the best deletion rule remains domain-dependent. 2 Problem Formulation We formulate lossy semantic text compression as a two-phase encode-decode problem. Given a source text T of length L and a target retention rate r_{\mathrm{keep}}\in(0,1) , the encoder produces a degraded representation \tilde{T} of length r_{\mathrm{keep}}\cdot L by strategically deleting components from T , and the decoder D_{\phi} reconstructs \hat{T}=D_{\phi}(\tilde{T}) from the degraded input alone. The effective character-level compression ratio is 1/r_{\mathrm{keep}} . At inference time, only \tilde{T} and lightweight strategy metadata need to be stored or transmitted; decoder weights and prompt templates are treated as shared side information. This setup is intentionally narrower than continuous latent semantic coding: our focus is on compression schemes whose intermediate representation remains text. We evaluate reconstruction quality primarily with BERTScore ( 29 ) , averaged over reconstructed chunks, and report ROUGE-L and Character Error Rate (CER) as complementary lexical-fidelity metrics. BERTScore appears in the main text; ROUGE-L and CER are reported in Appendix B and Appendix D . Because no single automatic metric fully captures factual faithfulness, we also report skeleton-level named-entity preservation and discuss explicit failure modes at low retention. 3 Methodology This section describes the full encode–decode pipeline. On the encoder side, strategic deletion compresses a source text into a textual skeleton under a retention budget. On the decoder side, LLM-based semantic reconstruction expands that skeleton into a fuller version of the source. Encoder: Structured Degradation Strategies We organize the encoder strategies into three levels of linguistic sophistication: Level 1 treats text as a character stream, Level 2 respects word boundaries and uses corpus-frequency statistics, and Level 3 uses a neural language model to estimate contextual predictability. Higher levels improve token selection at higher encoder cost, a trade-off quantified in Section 4.2 . Level 1: Character-Level Deletion

Read the complete text