SpeContext : Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs

Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai · 2025

Read SpeContext : Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs by Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00722.txt.

Introduction

Jiaming Xu and six coauthors address a cost problem in large language models in their 2025 paper, SpeContext. Reasoning models now write long step-by-step answers, which improves their work. The catch is memory: long outputs build a large key-value cache that stores past context for the model.

That cache becomes a bottleneck when such models run in real products, such as software agents. SpeContext uses speculative context sparsity, keeping only the parts of context that matter most. Guessing well about what can be thinned out could cut memory sharply. If quality holds, long reasoning becomes cheaper to serve at scale.

Catalog record

CREATORJiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-00722

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpeContext : Enabling Efficient Long-context Reasoning with Speculative Context Sparsity in LLMs Jiaming Xu, Jiayi Pan, Hanzhen Wang, Yongkang Zhou, Jiancai Ye, Yu Wang, Guohao Dai Abstract . As test-time scaling in large language model(LLM) reasoning has been proven effective in enhancing the model performance through step-by-step generation, this long-context generation incurs substantial Key-Value(KV) cache, posing a critical bottleneck for practical applications deployment( e.g. , Agents). While recent KV cache optimizations perform well in the long-context input scenario, the following problems remain unsolved if directly applied to long-context reasoning. (1) Time-consuming layer-wise retrieval operation . The retrieval operation, which selects the important KV pairs in each layer, brings the synchronization overhead that scales with model depth due to the data dependency, resulting in up to 60 % 60\% latency overhead. (2) Complete retention of the newly generated KV cache . Existing works designed for long-context input choose to retain the KV pair of newly generated tokens to avoid repeated, time-consuming processing on the KV cache, rendering them ineffective in long-context reasoning. (3) Performance degradation with a tiny increase in sequence length . Existing offloading strategies determined before inference cannot adapt to the increasing sequence length, resulting in > 80 % >80\% performance degradation with a tiny increase in sequence length. In this paper, we point out that the objective of the retrieval algorithms is to align with the LLM, which is similar to the objective of knowledge distillation in LLMs. We analyze the similarity in information focus between the distilled language model(DLM) and the original LLM from the perspective of information theory, and thus propose a novel paradigm that leverages a DLM as the retrieval algorithm . Based on the insight, we present SpeContext , an algorithm and system co-design for long-context reasoning. (1) At the algorithm level, SpeContext proposes lightweight retrieval head based on the head-level attention weights of DLM, achieving > 90 % >90\% parameters reduction by pruning the redundancy. (2) At the system level, SpeContext designs an asynchronous prefetch dataflow via the elastic loading strategy , effectively overlapping KV cache retrieval with the LLM computation. (3) At the compilation level, SpeContext constructs the theoretical memory model and implements an adaptive memory management system to achieve acceleration by maximizing GPU memory utilization. We deploy and evaluate SpeContext in two resource-constrained environments, cloud and edge. Extensive experiments show that, compared with the Huggingface framework, SpeContext achieves up to 24.89 × \textbf{24.89}\times throughput improvement in cloud and 10.06 × \textbf{10.06}\times speedup in edge with negligible accuracy loss, pushing the Pareto frontier of accuracy and throughput. Figure 1. (a)(b) Pareto frontiers on KV cache selection in long-context input and reasoning scenarios. 1. Introduction Figure 2. Overview of SpeContext . (a) Three challenges in existing algorithm paradigm in the long-context reasoning scenario. (b) Key Insight: Distilled language model exhibits similar information focus. (c) C ontributions from Section 4 to Section 6 Generative large language models(LLMs) mark a significant advancement in the pursuit of Artificial General Intelligence(AGI). Their successful application across various domains has greatly contributed to the rapid advancement of numerous downstream tasks ( e.g. , pharmaceutical ( 29 ) , finance ( 19 ) , and ecology ( 27 ) ), and their remarkable capabilities have attracted widespread attention, spurring the development of a new wave of LLM-based software applications ( e.g. , AI agents ( 14 ) ). As the scaling law gradually slows down, the test-time scaling ( 28 ) in LLM reasoning is emerging as a powerful tool in enhancing the model capabilities, especially in solving complex problems( e.g. , mission planning ( 14 ; 10 ; 44 ) and mathematical derivation ( 1 ) ), through step-by-step chain-of-thought generation ( 15 ) . Moreover, some latest works point out that as the length of the chain-of-thought reasoning increases, the LLM capabilities, especially the mission planning( e.g. , 8K reasoning length) and information search( 5M search length) in AI agents ( 44 ) , can be significantly improved. To effectively support long-context reasoning, LLM providers have enhanced the long-context processing capabilities of their LLMs during pretraining, with context windows of over 100K tokens becoming a common standard ( e.g. , Kimi K2 with 128K ( 35 ) and OpenAI o3 with 200K ( 11 ) ). Despite these algorithm advances, the computational and memory burden associated with Key-Value(KV) cache still prevents the efficient practical deployment. KV cache is a fundamental component in the LLM, which effectively reduces the computation by reusing past key-value pairs, but introduces significant memory overhead, which is proportional to the context length. Furthermore, during the autoregressive decoding, the generation of each new token requires reading the entire KV cache to compute attention weights, leading to severe latency overhead. For example, in the case of Llama3.1-8B ( 13 ) on a NVIDIA RTX 4090 GPU, generating a single token with a 16K context takes twice as long as generating a token with a 1k context, and theoretically generates 2GB of memory footprint for KV cache, which means that for an NVIDIA RTX 4090 GPU, only 3 requests can be processed in parallel at most shown in Figure 1 . For the LLM cloud service vendors, the longer response and limited throughput will translate to higher infrastructure costs ( e.g. , energy and hardware consumption) and suboptimal user experiences ( 16 ) . Consequently, many previous works have explored various techniques for KV cache optimization, particularly in resource-constrained environments by reducing the KV cache involved during inference, encompassing algorithm optimization ( e.g. , permanent eviction ( 6 ; 39 ) and dynamic selection ( 33 ; 24 ; 32 ) of KV cache), system enhancement ( e.g. , customized CUDA kernel design ( 33 ; 24 ; 32 ) ). These algorithms establish a paradigm centered on layer-wise retrieval operation during the decoding phase shown in Figure 2 (a). LLM inference can be divided into two phases, the prefill and decoding phase detailed in Section 2 . Most previous works ( 24 ; 32 ; 33 ) preprocess KV cache upon completion of the prefill phase, and the corresponding retrieval algorithms retrieve a subset of preprocessed KV cache for each generation during decoding phase. However, the core trade-off of this paradigm is its departure from mathematical equivalence. By selectively computing attention over a fraction of the context, these methods inherently introduce computational shortcuts that can lead to a degradation in model accuracy. Therefore, as illustrated in Figure 1 (a)(b), two Pareto frontiers in long-context input and reasoning scenarios are established, forcing a compromise between inference speed and model accuracy. Despite this paradigm performing well in the long-context input scenario, it still suffers from the following critical limitations during the decoding phase if directly applied in the long-context reasoning scenario. Challenge-1 : Time-consuming layer-wise retrieval operation. Figure 2 (a) shows that the algorithm paradigm needs to perform the retrieval over the KV cache and load the corresponding KV pairs based on the retrieval result before attention computation in each layer, resulting in the sequential dataflow due to data dependency. This serialization introduces substantial synchronization overhead, breaking the natural overlap between computation and memory access in the original pipeline. Furthermore, the retrieval operation is repeated in each layer

Read the complete text