ParallelComp: Parallel Long-Context Compressor for Length Extrapolation

Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, Ngai Wong · 2025

Read ParallelComp: Parallel Long-Context Compressor for Length Extrapolation by Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, Ngai Wong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14317.txt.

Introduction

Jing Xiong and co-authors present in 2025 ParallelComp, a parallel squeeze method that stretches model context past training length. Models trained on short text choke on very long inputs from memory caps and attention quirks. Their method needs no new training to reach far longer spans.

Input splits into chunks, and spare chunks plus off-topic tokens drop out on the fly. A parallel cache purge keeps memory in bounds on one large chip. Bias study shows sink, recency, and middle pulls act in distinct ways at long spans. An eight-billion model hits near top closed-model grades at long context.

Catalog record

CREATORJing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, Ngai Wong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-14317

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ParallelComp: Parallel Long-Context Compressor for Length Extrapolation Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, Ngai Wong Abstract Extrapolating ultra-long contexts (text length > 128 >128 K) remains a major challenge for large language models (LLMs), as most training-free extrapolation methods are not only severely limited by memory bottlenecks, but also suffer from the attention sink, which restricts their scalability and effectiveness in practice. In this work, we propose ParallelComp , a parallel long-context compression method that effectively overcomes the memory bottleneck, enabling 8B-parameter LLMs to extrapolate from 8K to 128K tokens on a single A100 80GB GPU in a training-free setting. ParallelComp splits the input into chunks, dynamically evicting redundant chunks and irrelevant tokens, supported by a parallel KV cache eviction mechanism. Importantly, we present a systematic theoretical and empirical analysis of attention biases in parallel attention—including the attention sink, recency bias, and middle bias—and reveal that these biases exhibit distinctive patterns under ultra-long context settings. We further design a KV cache eviction technique to mitigate this phenomenon. Experimental results show that ParallelComp enables an 8B model (trained on 8K context) to achieve 91.17 % 91.17\% of GPT-4’s performance under ultra-long contexts, outperforming closed-source models such as Claude-2 and Kimi-Chat. We achieve a 1.76x improvement in chunk throughput, thereby achieving a 23.50x acceleration in the prefill stage with negligible performance loss and pave the way for scalable and robust ultra-long contexts extrapolation in LLMs. We release the code at https://github.com/menik1126/ParallelComp . † † affiliationnotice: Equal contribution 1 Introduction Extrapolating long-contexts is a core capability of large language models (LLMs) ( Achiam et al., 2023 ; Touvron et al., 2023 ; Dubey et al., 2024 ) . Achieving this requires effective mechanisms for modeling positional relationships over extended sequences. Rotary position embedding (RoPE) ( Su, 2023 ) are commonly employed in LLMs due to their ability to efficiently encode relative positional information. However, extrapolating to lengths beyond the training range during inference remains a significant challenge. While retraining or fine-tuning the entire model is one possible solution, it is not always feasible, particularly in resource-constrained environments ( Press et al., 2022 ; Chi et al., 2022 ; Peng et al., 2023 ; Fu et al., 2024 ) . This limitation prevents RoPE-based models from generalizing to longer sequences without additional training, making them less adaptable to scenarios requiring inference on inputs of unforeseen lengths. Figure 1 : Upper: Distribution of two types of attention. Lower: Heatmaps of the two types of attention. The attention head shown here represents the distribution change between normal attention and parallel attention for the 21st head in layer 1, using the Llama-2-7b-chat-hf model. To address this, many researchers explore training-free approaches for context length extrapolation. One prominent direction involves incorporating NTK-based methods ( LocalLLaMA, 2023b ; LocalLLaMA, 2023a ; bloc97, 2023 ) , which distinguish dimensions of different frequencies by their wavelengths and apply tailored interpolation strategies to address the extrapolation challenge of position encoding. Another promising strategy leverages text chunking techniques ( An et al., 2024 ; Xiao et al., 2024 ; Ratner et al., 2022 ; Zhu et al., 2024 ) , which typically reuse position encodings across chunks to enable chunk-wise length extrapolation without retraining, thereby improving the model’s ability to generalize to longer contexts. However, both approaches suffer from the attention sink phenomenon ( Liu et al., 2024 ; Xiao et al., 2023 ; Han et al., 2024 ; Gu et al., 2024 ) , where high attention scores tend to be assigned to the first few tokens or the last few tokens in the input sequence. Figure 1 illustrates this phenomenon and compares the attention distributions between standard and parallel attention mechanisms. In the upper part of Figure 1 , we observe the distribution of attention scores across tokens. In the standard attention distribution (left), corresponding to the heatmaps (c) below the image, attention is disproportionately focused on the first and last few tokens. In contrast, the parallel attention distribution (right), shown in the heatmaps (d) below the image, attempts to distribute attention more evenly. However, it still shows noticeable concentrations in certain regions and exhibits a multi-peak distribution, indicating that the nature of the attention sink differs fundamentally between the two mechanisms. The phenomenon within the length extrapolation mechanism in the parallel attention mechanism remains unexplored. This paper focuses on addressing the memory limitations encountered during length extrapolation and provides a detailed analysis of the unique attention sink that emerges in parallel attention. Specifically, we extrapolate the length by chunking the input and reusing position encodings, applying parallel compression of the KV cache of the chunks and tokens to resolve memory bounds, and exploring the unique phenomenon of the attention sink in parallel attention while attempting to adopt a token eviction strategy to mitigate this bias. To promote the understanding of the effects of special attention sink on parallel attention, we propose the following questions in this paper: Q1: What types of attention patterns can be summarized? Q2: Is there any difference between the attention bias in parallel attention and the attention bias in classical attention? Q3: Can the calibration strategy alleviate attention bias? Q4: Does our parallel compression strategy effectively support length extrapolation? To address the above questions, we propose ParallelComp , a parallel long-context compression method to extrapolate length. While maintaining high throughput, we extrapolate the length from 8K to 128K on a single GPU, with almost no performance loss. Overall, our contributions are as follows: • We propose ParallelComp , a novel training-free method that enables efficient length extrapolation for LLMs — scaling from 8K to up to 128K tokens on a single A100 80GB GPU — by addressing attention biases through an effective attention calibration strategy. • To overcome limitations in parallel attention, we introduce a chunk eviction mechanism and parallel KV cache eviction, allowing processing of contexts beyond 128K tokens and achieving a 1.76x throughput improvement and a 23.50x speedup in the prefilling stage, with negligible performance loss. • Experiments demonstrate that ParallelComp achieves 91.17% of GPT-4’s performance on ultra-long context tasks using an 8B model trained on only 8K-length context, outperforming strong proprietary models such as Claude-2 and Kimi-Chat. Figure 2 : Overview of ParallelComp . Parallel Attention – The input sequence is split into multiple chunks based on the model’s maximum context length. Each chunk undergoes local attention computation independently, and the self-information score of the query is calculated. Parallel KV Cache Eviction – Based on the self-information score, low-score tokens (marked in yellow, R^{h}_{L} ) and attention bias tokens (marked in red, R^{h}_{H} ) are selectively evicted to optimize memory usage and attention bias. Global Attention – The remaining KV caches are ranked by self-information, and less relevant chunks are discarded. The selected chunks are then concatenated, and a global attention operation is applied to ensure comprehensive information aggregation before the final autoregressive decoding stage. 2 Related Work Position encoding. Existing absolute position encoding (APE) ( Vaswani, 2017 ; Devlin, 2018 ) incorporates either fixed or learnable position encodings into input representations through vector addition. However, APE faces challenges when dealing with long-contexts. To overcome these limitations, relative position encoding (RPE)

Read the complete text