PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression

Lizhe Chen, Binjia Zhou, Yuyao Ge, Jiayi Chen, Shiguang Ni · 2025

Read PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression by Lizhe Chen, Binjia Zhou, Yuyao Ge, Jiayi Chen, Shiguang Ni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-16574.txt.

Introduction

Lizhe Chen and four co-authors present a 2025 method for shrinking the prompts fed to large language models. Long prompts cost money and slow these models down, yet common ways to trim them use rough cut rules or rewrites that ignore how the model itself works. The team's method, called PIS, instead scores each word by how much the model attends to it.

The method trims at two levels. A small trained network picks which words to keep, guided by the model's own attention scores. A second step samples whole sentences by a chance rule that favors the ones that matter most. Tests across several task sets gave top scores for this kind of trimming. The authors also found that the trimmed prompts, being better shaped, helped the model reason faster.

Catalog record

CREATORLizhe Chen, Binjia Zhou, Yuyao Ge, Jiayi Chen, Shiguang Ni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-16574

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression Lizhe Chen, Binjia Zhou, Yuyao Ge, Jiayi Chen, Shiguang Ni Abstract Large language models (LLMs) have achieved remarkable progress, demonstrating unprecedented capabilities across various natural language processing tasks. However, the high costs associated with such exceptional performance limit the widespread adoption of LLMs, highlighting the need for prompt compression. Existing prompt compression methods primarily rely on heuristic truncation or abstractive summarization techniques, which fundamentally overlook the intrinsic mechanisms of LLMs and lack a systematic evaluation of token importance for generation. In this work, we introduce Prompt Importance Sampling (PIS), a novel compression framework that dynamically compresses prompts by sampling important tokens based on the analysis of attention scores of hidden states. PIS employs a dual-level compression mechanism: 1) at the token level, we quantify saliency using LLM-native attention scores and implement adaptive compression through a lightweight 9-layer reinforcement learning network; 2) at the semantic level, we propose a Russian roulette sampling strategy for sentence-level importance sampling. Comprehensive evaluations across multiple domain benchmarks demonstrate that our method achieves state-of-the-art compression performance. In particular, our framework improves reasoning efficiency to some extent through optimized context structuring. This work advances prompt engineering by offering both theoretical grounding and practical efficiency in context management for LLMs. 1 Introduction Large language models (LLMs) have made remarkable progress, showing unprecedented capabilities across various natural language processing tasks ( Vaswani 2017 ; Devlin et al. 2018 ; Brown et al. 2020 ; Ouyang et al. 2022 ) . However, the iterative generation process introduces critical computational bottlenecks that disproportionately affect system performance - particularly in reasoning and planning tasks. As token sequences grow through successive iterations, memory consumption exhibits quadratic scaling relative to sequence length, while inference latency increases linearly. This fundamental conflict between the demand for deeper reasoning capabilities and practical computational constraints underscores the urgent need for efficient prompt compression methodologies. Such techniques require an optimal balance between semantic preservation and token reduction, maintaining critical informational fidelity while significantly decreasing computational overhead. Figure 1: Compared to methods that train a specific model based on the LLM input and its compressed version, our approach achieves ultra-low-loss and high-performance compression through importance sampling at two levels, which only need a compact 9-layer RL policy network. Current approaches primarily adapt conventional text summarization paradigms, which can be categorized into two groups: 1) generative compression methods ( e.g. , LLMLingua Jiang et al. 2023a ), which employ auxiliary language models for prompt rewriting, and 2) heuristic pruning techniques ( e.g. , Selective Context Li et al. 2023 ), which eliminate tokens based on surface-level metrics. Although these strategies offer some solutions, they fundamentally overlook two critical aspects of LLM operation: (a) the excessive computational cost introduced by the reliance on external generative models, which significantly increases training and inference overhead, and (b) the failure of heuristic pruning to account for the internal mechanisms of the LLM, resulting in token sequences that may not align with the model’s optimal representation of relevant information. These issues result in current methods being either not efficient or not effective. This work addresses two fundamental research questions through a unified theoretical and methodological framework: • How to achieve higher compression quality without relying on generative models? • How to formalize prompt compression in alignment with LLMs’ computational mechanisms? To resolve these challenges, we propose Prompt Importance Sampling (PIS), a novel compression methodology that synergizes LLM properties with importance sampling. First, we establish a measure-theoretic foundation for prompt compression, formalizing the relationship between token importance and the distribution of LLM attention scores. Building on this theoretical framework, we develop a dual-level compression architecture: 1) at the token level, we quantify saliency through attention score analysis and implement adaptive pruning via a lightweight 9-layer RL network; 2) at the sentence level, we devise a Russian roulette mechanism for probabilistic redundancy reduction across semantic units. Figure 1 shows the advantages of our method over previous methods. Comprehensive evaluations across multi-domain benchmarks demonstrate that PIS significantly outperforms existing baselines. Our method achieves a 15% performance improvement at equivalent compression ratios by strategically preserving attention-critical information. Our framework, relying solely on native attention computations and a compact RL module, reduces inference overhead by 38% compared to strong baseline compression approaches. Notably, the optimized context structuring serendipitously enhances reasoning efficiency, yielding a 5% accuracy improvement on downstream tasks when using compressed prompts versus original inputs. The source code is available as open-source on Github . Our principal contributions are threefold: • We present a measure-theoretic formulation of prompt compression that interprets LLM attention score allocation as a measurable function, providing theoretical foundations for prompt context optimization. • We propose PIS, a novel prompt compression approach that performs importance sampling at both the token and sentence levels, achieving a balance between efficiency and effectiveness. • A comprehensive experimental evaluation demonstrates that our approach, by incorporating the attention mechanism of LLMs, achieves higher compression quality and improves compression efficiency. 2 Related Works 2.1 Prompt Engineering Prompt engineering has been widely applied to enhance the performance of large language models (LLMs) for various tasks. Significant progress has been made in improving reasoning abilities through techniques such as Wei et al. 2022 , which introduces step-by-step reasoning prompts. Subsequent refinements to the chain-of-thought (CoT) paradigm have been proposed in Yao et al. 2024 ; Shao et al. 2024 ; Suzgun and Kalai 2024 , demonstrating continuous improvements in problem-solving capabilities. The integration of Retrieval-Augmented Generation (RAG) frameworks for information retrieval has been explored in Ge et al. 2024b ; Zhang et al. 2024a ; Jiang et al. 2024a ; Ng et al. 2024 , while memory enhancement through structured prompting is investigated in Park et al. 2023 ; Liu et al. 2023 ; Li et al. 2024 ; Zeng et al. 2024 ; Hou et al. 2024 . Applications in knowledge editing are examined in Muric et al. 2024 ; Chen et al. 2024 ; Zhang et al. 2024b ; Ge et al. 2024a , with security-focused adaptations discussed in Kumar et al. 2023 ; Zheng et al. 2024 . These advancements highlight the critical role of compression in optimizing prompt engineering efficiency. 2.2 Text Compression Techniques Current prompt compression methods are primarily derived from two paradigms in the text summarization domain. Generative compression methods are closely related to traditional text summarization tasks, as they involve content distillation and regeneration. This enables the cross-application of techniques between the two domains. Zhang et al. 2020 ; Chevalier et al. 2023 and other existing approaches typically employ auxiliary models for content rewriting, where See et al. 2017 pioneered sequence-to-sequence frameworks with copy mechanisms, and Guo et al. 2018 advanced the field by employing GAN-style training with generator-discriminator pairs. However, these methods incur substantial computational overhead from the use of external models, significantly limiting their practical utility for LLM deployment. Heuristic pruning methods focus on token elimination using surface-level metrics. Mihalcea and Tarau 2004 and other early work rank sentences via graph centrality, a strategy later adapted for prompting through similarity thresholds. Zhou et al. 2018 ; Jiang et al. 2023a ; Pan et al. 2024b ; Yang and Xue 2022 and other advancements incorporate sentence

Read the complete text