PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference
Weisheng Jin, Maojia Song, Tej Deep Pala, Yew Ken Chia, Amir Zadeh, Chuan Li, Soujanya Poria · 2025
Read PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference by Weisheng Jin, Maojia Song, Tej Deep Pala, Yew Ken Chia, Amir Zadeh, Chuan Li, Soujanya Poria (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23274.txt.
Introduction
Weisheng Jin and six co-authors trimmed in 2025 the work language models do while reading long prompts. Most of a prompt is filler for any given question, yet standard models grind through every token at every layer. Their method, PromptDistill, scores tokens early by how much the question attends to them.
Only the informative tokens keep their full inner state into the deep layers; the rest retire early. Unlike rivals, the choice adapts to each question rather than fixing a slice of the prompt. No retraining is needed. Cheaper reading means longer documents fit smaller budgets.
Catalog record
| CREATOR | Weisheng Jin, Maojia Song, Tej Deep Pala, Yew Ken Chia, Amir Zadeh, Chuan Li, Soujanya Poria |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23274
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference Weisheng Jin, Maojia Song, Tej Deep Pala, Yew Ken Chia, Amir Zadeh, Chuan Li, Soujanya Poria Abstract As large language models (LLMs) tackle increasingly complex tasks and longer documents, their computational and memory costs during inference become a major bottleneck. To address this, we propose PromptDistill, a novel, training-free method that improves inference efficiency while preserving generation quality. PromptDistill identifies and retains the most informative tokens by leveraging attention interactions in early layers, preserving their hidden states while reducing the computational burden in later layers. This allows the model to focus on essential contextual information without fully processing all tokens. Unlike previous methods such as H2O and SnapKV, which perform compression only after processing the entire input, or GemFilter, which selects a fixed portion of the initial prompt without considering contextual dependencies, PromptDistill dynamically allocates computational resources to the most relevant tokens while maintaining a global awareness of the input. Experiments using our method and baseline approaches with base models such as LLaMA 3.1 8B Instruct, Phi 3.5 Mini Instruct, and Qwen2 7B Instruct on benchmarks including LongBench, InfBench, and Needle in a Haystack demonstrate that PromptDistill significantly improves efficiency while having minimal impact on output quality compared to the original models. With a single-stage selection strategy, PromptDistill effectively balances performance and efficiency, outperforming prior methods like GemFilter, H2O, and SnapKV due to its superior ability to retain essential information. Specifically, compared to GemFilter, PromptDistill achieves an overall 1 % 1\% to 5 % 5\% performance improvement while also offering better time efficiency. Additionally, we explore multi-stage selection, which further improves efficiency while maintaining strong generation performance. These results demonstrate the adaptability of LLM hidden representations and provide valuable insights into optimizing inference for long-context scenarios. We have made our implementations publicly available 1 1 1 https://github.com/declare-lab/PromptDistill . 1 Introduction Large language models (LLMs), such as GPT, LLama, Gemini, Mistral, and Claude ( Brown et al., 2020 ; Grattafiori et al., 2024 ; Team et al., 2024 ; Jiang et al., 2023 ; Anthropic, 2024 ) have achieved remarkable success across various natural language processing (NLP) tasks, demonstrating strong generalization and reasoning abilities. However, their widespread deployment is often hindered by the high computational and memory costs, particularly during inference with long context. As context lengths continue to grow in modern LLM applications, efficiently handling extended inputs while preserving generation quality has become a critical challenge. Some approaches, such as MCSD and Mamba ( Yang et al., 2024b ; Gu & Dao, 2024 ) , attempt to mitigate this issue by compressing internal states into a single vector. While these methods reduce processing complexity from quadratic to linear, they often suffer from performance degradation and scalability challenges, as their designs sacrifice key Transformer advantages. Other methods modify existing architectures to optimize inference efficiency, requiring extensive retraining to adapt model parameters to the new structure ( Wang et al., 2024 ; Gao et al., 2024 ) . However, such approaches incur high training costs and risk compromising the original model’s capabilities. Methods like H2O and Snap ( Zhang et al., 2023 ; Li et al., 2024b ) require no extra training and can be directly applied to existing models for long-context compression. However, they still process all prompt tokens through all model layers once before compression, resulting in significant computational overhead. GemFilter ( Shi et al., 2024 ) , which also requires no extra training, mitigates this issue by processing only part of the layers for all prompt tokens before selecting the most important ones. It then re-runs the model using only the selected tokens for full-layer processing. While this reduces computational cost and GPU memory usage, it also discards all unselected tokens’ hidden representations, retaining only token IDs after early-layer processing. This may lead to information loss, disrupt context structures, and ultimately impact performance, especially when the selection process is imperfect or the number of retained tokens is limited. We propose PromptDistill, a training-free method to address these limitations, achieving high efficiency while maintaining performance. First, we process a few layers for all tokens as in traditional transformers. Then, we select the most important prompt tokens based on the dot product between the query of the last prompt token and the key of each token. This selection criterion is similar to GemFilter, but instead of reprocessing the selected tokens from the first layer, we retain their hidden states after the selection layer and continue processing them through the remaining layers. The selected tokens’ hidden states already capture context information from earlier layers’ self-attention, allowing us to preserve more comprehensive context and the model’s original structure without additional computation. A potential efficiency issue is that during generation, each new token attends to all prompt tokens in the initial layers, leading to high time and memory costs. To mitigate this, we introduce cache truncation, clearing the keys and values of unselected tokens in the key-value cache for layers before selection. This improves both time and memory efficiency while maintaining performance (as shown in section 2.3 and section 3.3 ). Our method thus achieves slightly better computational efficiency than GemFilter since we avoid recomputing the selected tokens in the first few layers. Unlike GemFilter, which can select important tokens only once in a specific layer, PromptDistill allows for multi-stage selection across layers. Early layers struggle to select tokens effectively ( Voita et al., 2019 ) , but selecting more tokens in these layers can still enhance efficiency without sacrificing performance. A combination of selecting many tokens in the early layers and fewer in later layers strikes a balance between efficiency and effectiveness. In our experiments, we apply PromptDistill to LLMs such as LLaMA-3.1, Qwen2, and Phi3 ( Grattafiori et al., 2024 ; Yang et al., 2024a ; Abdin et al., 2024 ) and evaluate on reliable benchmarks like Longbench, Infbench, and Needle in a Haystack ( Bai et al., 2024 ; Zhang et al., 2024a ; Nelson et al., 2024 ) . These extensive experiments clearly show that our method outperforms baseline methods, including full attention, GemFilter, SnapKV, and H2O ( Shi et al., 2024 ; Li et al., 2024b ; Zhang et al., 2023 ) . In particular, compared to GemFilter, our method improves time efficiency and achieves better performance. Additionally, we provide a comprehensive analysis of other aspects of our method, demonstrating how multi-stage selection improves efficiency while preserving performance. Our contributions are summarized as: • We propose PromptDistill, a training-free, generic modification for LLMs that significantly boosts efficiency while maintaining comparable performance to full attention. It selects key tokens and retains their hidden states in intermediate layers to create an informative compression of the entire prompt. • We conduct extensive zero-shot experiments with single-stage selection to evaluate PromptDistill’s ability to balance effectiveness and efficiency compared to several well-designed baselines. • With multi-stage selection, PromptDistill further reduces computational costs without sacrificing performance. The experiments also provide insights into token representativeness across different layers. 2 Method 2.1 Background To provide a clear background for our approach, PromptDistill, discussed in section 2.2 , we first define the relevant preliminaries and introduce the token selection method used in our approach.