SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning
Lingkun Long, Ruibin Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang · 2025
Read SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning by Lingkun Long, Ruibin Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06447.txt.
Introduction
Lingkun Long and co-authors present SlimInfer in 2025, a method that speeds up language models working on very long inputs. Long prompts are costly because most systems still process every token at every layer.
The key insight is that information from key tokens spreads across the whole sequence as layers pass. After that spread, extra tokens can be dropped mid-network without losing meaning. SlimInfer prunes tokens layer by layer, guided by fine-grained importance. Its cache manager fetches needed blocks ahead of time without predictors. On a standard model it more than doubled the speed to first output, with no loss on a long-context benchmark.
Catalog record
| CREATOR | Lingkun Long, Ruibin Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06447
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning Lingkun Long, Ruibin Yang, Yushi Huang, Desheng Hui, Ao Zhou, Jianlei Yang Abstract Long-context inference for Large Language Models (LLMs) is heavily limited by high computational demands. While several existing methods optimize attention computation, they still process the full set of hidden states at each layer, limiting overall efficiency. In this work, we propose SlimInfer, an innovative framework that aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass. Our key insight is an information diffusion phenomenon : As information from critical tokens propagates through layers, it becomes distributed across the entire sequence. This diffusion process suggests that LLMs can maintain their semantic integrity when excessive tokens, even including these critical ones, are pruned in hidden states. Motivated by this, SlimInfer introduces a dynamic fine-grained pruning mechanism that accurately removes redundant tokens of hidden state at intermediate layers. This layer-wise pruning naturally enables an asynchronous KV cache manager that prefetches required token blocks without complex predictors, reducing both memory usage and I/O costs. Extensive experiments show that SlimInfer can achieve up to 2.53 × \mathbf{2.53\times} time-to-first-token (TTFT) speedup and 1.88 × \mathbf{1.88\times} end-to-end latency reduction for LLaMA3.1-8B-Instruct on a single RTX 4090, without sacrificing performance on LongBench. Our code will be released upon acceptance. 1 Introduction Large Language Models (LLMs) have shown strong performance in long-context tasks such as summarization ( Zhang et al. 2020 ; Kryściński et al. 2022 ) , multi-document question answering ( Yang et al. 2018 ) , and retrieval from extended inputs ( Bai et al. 2024 ) . Scaling to longer sequences not only enables more complex reasoning, but also introduces substantial computational and memory overhead as the context length increases ( Fu 2024 ) . Figure 1: Accuracy vs. inference efficiency across different acceleration approaches on LongBench ( Bai et al. 2024 ) for LLaMA-3.1-8B-Instruct ( Grattafiori et al. 2024 ) . During the prefill stage, the self-attention mechanism ( Vaswani et al. 2017 ) incurs quadratic time complexity with respect to the sequence length, making it a major source of latency in long-context scenarios. At the same time, the Key-Value (KV) cache grows linearly with input length, leading to substantial GPU memory consumption. To mitigate these issues, numerous token pruning methods have been proposed. However, existing token pruning methods face several critical limitations. Some works ( Zhang et al. 2023 ; Xiao et al. 2024b ; Li et al. 2024 ; Yang et al. 2024 ; Wang et al. 2025 ; Cai et al. 2025 ; Hao et al. 2025 ; Nguyen et al. 2025 ) focus primarily on optimizing the decoding phase, offering minimal improvements to the critical Time-To-First-Token (TTFT). In addition, their token eviction strategies often lead to accuracy degradation due to the removal of contextually important information. Other approaches ( Lai et al. 2025 ; Jiang et al. 2024 ) extend support to both prefill and decoding phases by sparsifying the attention pattern ( Deng et al. 2025 ) . Nevertheless, they process the full sequence of hidden states at every layer, leaving non-attention components like Feed-Forward Networks (FFNs) unoptimized and limiting overall acceleration. Memory efficiency presents an additional challenge. Dynamic token pruning methods ( Fu et al. 2024 ) retain the entire KV cache in the GPU, leading to excessive memory consumption and limited scalability for longer sequences. To alleviate this, some systems offload the KV cache to the CPU ( Tang et al. 2024 ) , which reduces GPU pressure ( Gong et al. 2024 ; Huang et al. 2024 ) but introduces significant I/O latency. More recent designs attempt to prefetch KV segments to overlap data transfer and computation ( Lee et al. 2024 ; Yang et al. 2025 ) . However, these approaches often rely on predictor-based mechanisms, introducing additional overhead and complexity. Therefore, existing token pruning methods still struggle to simultaneously optimize inference speed ( Jiang et al. 2024 ; Huang et al. 2025c ) , memory usage ( Xiao et al. 2024b ; Huang et al. 2025a ) , and model performance ( Huang et al. 2025b ; Wnag et al. 2024 ) . In this paper, we propose SlimInfer , a framework designed to accelerate inference by dynamically pruning less critical prompt tokens during the forward pass. Our method builds on a key insight we term the information diffusion phenomenon : As information from critical tokens propagates through the layers of an LLM, it becomes progressively distributed across other token representations. This diffusion process suggests that LLMs can maintain their semantic integrity even when excessive tokens are pruned in hidden states, including those essential initially. Motivated by this insight, SlimInfer introduces a dynamic layer-wise pruning to the hidden states across intermediate layers, progressively reducing computational workload. To preserve essential semantic information while maximising efficiency, we further introduce a fine-grained, block-wise importance evaluation that retains only the contextually relevant tokens. This pruning mechanism works in tandem with an asynchronous KV cache manager, which exploits the determinism of pruning decisions to enable predictor-free prefetching and efficient GPU memory management. We conduct extensive experiments on LLaMA-3.1-8B-Instruct ( Grattafiori et al. 2024 ) and Qwen2.5-7B-Instruct ( Qwen et al. 2025 ) . As shown in Figure 1 , SlimInfer can achieve up to 2.53\times time-to-first-token (TTFT) speedup and a 1.88\times end-to-end latency reduction on a single NVIDIA RTX 4090 GPU. It also maintains near-lossless accuracy drops on the LongBench ( Bai et al. 2024 ) . 2 Related Works 2.1 Token Pruning Token pruning methods aim to reduce inference overhead by selectively removing less critical tokens from computation or memory. Many approaches target GPU memory reduction by maintaining a fixed-size KV cache. StreamingLLM ( Xiao et al. 2024b ) retains initial tokens (attention sinks) and a sliding window of recent tokens, but discards intermediate ones. H2O ( Zhang et al. 2023 ) proposes a heavy-hitter oracle that evicts tokens with low cumulative attention scores. Similarly, SnapKV ( Li et al. 2024 ) uses the local context of a prompt to predict and retain important tokens for future generation steps. LazyLLM ( Fu et al. 2024 ) introduces dynamic pruning based on token importance, but still retains most KV entries in GPU memory, limiting its scalability to longer contexts. A primary limitation of these methods is their irreversible token eviction, which permanently removes KV entries from GPU memory. This permanent removal can lead to significant accuracy degradation, particularly in complex tasks that rely on long-range dependencies scattered throughout the context. Unlike prior methods that irreversibly discard evicted tokens, SlimInfer offloads currently irrelevant tokens ( i.e. , pruned tokens) to CPU memory instead of discarding them, significantly improving performance and reducing GPU memory usage. Other methods focus on accelerating computation by inducing sparsity in the attention map. FlexPrefill ( Lai et al. 2025 ) , SpargeAttn ( Zhang et al. 2025b ) adopt block-level heuristics by constructing representative vectors for token chunks, enabling coarse-grained attention skipping. In contrast, MInference ( Jiang et al. 2024 ) predicts structured sparse patterns based on partial attention observations. However, they still compute over the full sequence of hidden states at every layer. As a result, non-attention components like