Beyond Position Bias : Shifting Context Compression from Position-Driven to Semantic-Driven

Jiwei Tang, Zhijing Huang, Xinyu Zhang, Chen Jason Zhang, Jianxing Yu, Libin Zheng, Rui Meng, Jian Yin · 2026

Read Beyond Position Bias : Shifting Context Compression from Position-Driven to Semantic-Driven by Jiwei Tang, Zhijing Huang, Xinyu Zhang, Chen Jason Zhang, Jianxing Yu, Libin Zheng, Rui Meng, Jian Yin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-09463.txt.

Introduction

Tang and seven colleagues propose SeCo, Semantic Consistency Context Compression, a method moving compression from position-led to meaning-led. Current soft prompt compression relies on tunable tokens at fixed positions or groups tokens by layout, causing shaky results and semantic fragmentation. SeCo instead anchors compression in semantic space, picking query-relevant tokens as centres and merging the rest by consistency weighting.

That design keeps semantic consistency while removing position bias. Wide experiments on 14 benchmarks across two backbone models show SeCo ahead on downstream tasks and run speed, with toughness in unseen domains.

Catalog record

CREATORJiwei Tang, Zhijing Huang, Xinyu Zhang, Chen Jason Zhang, Jianxing Yu, Libin Zheng, Rui Meng, Jian Yin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.09463

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Position Bias : Shifting Context Compression from Position-Driven to Semantic-Driven Jiwei Tang, Zhijing Huang, Xinyu Zhang, Chen Jason Zhang, Jianxing Yu, Libin Zheng, Rui Meng, Jian Yin Abstract Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks. However, their deployment in long-context scenarios faces high computational overhead and information redundancy. While soft prompt compression has emerged as a promising way to mitigate these costs by compressing sequences into compact embeddings, existing paradigms remain fundamentally constrained by position bias : they primarily rely on learnable tokens insertion at fixed positions or group tokens according to their physical token layout, thereby inducing performance instability and semantic fragmentation. To overcome this bottleneck, we propose Se mantic Co nsistency Context Compression ( SeCo ), a method that shifts context compression from position-driven to semantic-driven . Rather than constraint by physical token layout, SeCo dynamically anchors compression directly in the semantic space by selecting query-relevant tokens as semantic centers and aggregating remaining tokens via consistency-weighted merging. This design inherently preserves semantic consistency while eliminating position bias. Extensive experiments on 14 benchmarks across two backbone models demonstrate that SeCo consistently shows superiority in downstream tasks, inference latency, and out-of-domain robustness. The code is available at https://anonymous.4open.science/r/seco-EE5E . 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks, catalyzing a paradigm shift in Natural Language Processing (NLP) [ 59 , 10 , 52 ] . Real-world applications increasingly demand longer prompt inputs, such as Retrieval-Augmented Generation (RAG) [ 29 ] , In-Context Learning (ICL) [ 11 ] , and the deployment of LLMs as agents for long-horizon task planning [ 46 ] . Despite the enhanced contextual coverage afforded by long sequences, deploying LLMs directly on such inputs faces (1) significant information redundancy [ 36 , 22 , 17 , 50 , 51 ] and (2) high computational load caused by Transformer’s quadratic time complexity [ 55 , 48 , 49 ] on its self attention mechanism, limiting their real-world utility. (a) Positional Encoding Bias (b) Group-based Position Bias (c) SeCo Figure 1 : Comparison between existing methods and SeCo. (a) Performance Instability. Learnable token insertion method, such as ICAE [ 17 ] , exhibit pronounced sensitivity to the position bias factor \beta , which controls the positional encoding of learnable tokens (as detailed in Figure 5 ). Consequently, F1 scores fluctuate substantially across various benchmarks under different \beta configurations. (b) Semantic Fragmentation. Token A_{1} in Group A is semantically highly correlated with tokens B_{1} and B_{2} in Group B. However, position-based grouping rigidly partitions these semantically related tokens (highlighted in red boxes) into separate compression groups. (c) Semantic Consistency Compression (SeCo). SeCo selects the top- k tokens most relevant to the query as semantic centers. Subsequently, remaining tokens are dynamically assigned and merged directly in the semantic space based on query-centric semantic consistency. Prompt compression [ 31 ] has emerged as a promising way to shorten sequence lengths and mitigate redundancy. Existing methods can be mainly categorized into hard prompt compression [ 22 , 40 , 50 , 6 , 38 , 20 , 35 ] and soft prompt compression [ 17 , 62 , 32 , 3 , 9 , 43 , 51 , 48 , 49 ] . Hard compression methods typically discard non-selected tokens entirely, failing to leverage secondary contextual information and compromising semantic coherence, which limits information utility [ 64 ] . In contrast, soft prompt compression exploits inherent redundancy in the semantic space [ 15 , 1 ] by learning compact soft embeddings, typically preserving richer semantic information, especially under high compression rates [ 51 ] . However, we identify a critical bottleneck: existing soft compression methods, which can be categorized into learnable token insertion [ 17 , 62 , 32 , 3 , 9 , 43 ] and position-based grouping [ 2 , 51 , 48 , 49 ] , rely heavily on positional priors, introducing position bias . Specifically, current soft prompt compression methods suffer from two systematic forms of position bias : (1) Positional encoding bias. As illustrated in Figure 1(a) , methods employing learnable token insertion (e.g., ICAE [ 17 ] ) exhibit significant instability in downstream tasks depending on the Positional Encodings (PEs) assigned to these tokens. This sensitivity to perturbations in PEs [ 55 , 45 ] aligns with recent observations in [ 65 ] . (2) Group-based position bias. The principle of semantic consistency posits that semantically coherent vector structures facilitate more effective compression [ 18 , 8 , 12 ] . However, position-based grouping methods rely on the physical token layout , potentially partitioning semantically related tokens into distinct compression groups (see Figure 1(b) ). This induces semantic fragmentation, thereby violating semantic consistency. Furthermore, due to the inherent sparsity of task-relevant information [ 30 , 22 , 50 ] , the correlation between such derived groups and the query is often highly imbalanced (i.e., certain groups are entirely irrelevant to the query). These findings collectively highlight a core limitation: over-reliance on positional priors constrains compression effectiveness. This motivates our research: How can we mitigate the negative effects of position bias in soft prompt compression? To address this, we draw inspiration from cognitive science: human cognition is fundamentally organized as a semantic network rather than a linear sequence, where concepts are linked by meaning rather than physical proximity [ 7 , 41 ] . Motivated by this, we shift the compression paradigm from position-driven to semantic-driven , proposing the Se mantic Consistency Co ntext Compression ( SeCo ) framework (Figure 1(c) ). SeCo eliminates the need for injecting learnable tokens at fixed positions, instead modeling context structure directly within the semantic space to avoid positional encoding bias . To resolve semantic fragmentation caused by group-based postion bias, SeCo reconstructs the grouping logic guided by query semantics: we first compute the relevance between context tokens and the query to identify the top- k most relevant tokens as semantic centroids. Subsequently, remaining tokens are dynamically assigned to these centroids based on semantic consistency, and compressed representations are generated via consistency-weighted aggregation within each center. This process is driven by semantic relevance rather than physical token layout; each token autonomously identifies its suitable compression group directly in the semantic space, effectively mitigating group-based position bias. Our contributions are summarized as follows: (1) We identify and analyze position bias as a bottleneck that fundamentally limits existing soft prompt compression methods. (2) We propose SeCo, a semantic-driven compression framework designed to mitigate position bias and enhance semantic consistency. (3) We conduct extensive experiments across two backbone models (i.e., LLaMA and Qwen) and 14 benchmarks (e.g., long context question-answering and summarization), including both in-domain and out-of-domain evaluations. Experiment results demonstrate that SeCo consistently delivers superior performance in terms of downstream tasks, computational efficiency, and robustness. 2 Related Work Hard Prompt Compression. Hard prompt compression aims to reduce input length by explicitly shortening the original prompt, either through removing less informative tokens or generating condensed textual summaries. The resulting compressed prompt remains in the form of discrete natural language. Existing approaches can be broadly grouped into four categories [ 51 ] . (1) Perplexity-based methods estimate token importance according to perplexity and

Read the complete text