Position Bias Mitigates Position Bias: Mitigate Position Bias Through Inter-Position Knowledge Distillation
Yifei Wang, Feng Xiong, Yong Wang, Linjing Li, Xiangxiang Chu, Daniel Dajun Zeng · 2025
Read Position Bias Mitigates Position Bias: Mitigate Position Bias Through Inter-Position Knowledge Distillation by Yifei Wang, Feng Xiong, Yong Wang, Linjing Li, Xiangxiang Chu, Daniel Dajun Zeng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15709.txt.
Introduction
Yifei Wang and five co-authors tackle position bias in long texts, in a paper from two thousand twenty-five. Models often read the start or end of a long text well, but miss facts in the middle. Past fixes changed the model build or needed costly training on huge sets of text.
Their method, Pos2Distill, moves skill from strong spots to weak spots inside the same model. One form fits search tasks, and the other fits tasks that need step-by-step thought. Both rest on the same thought: use the gap itself to close the gap. Tests show more even scores across all spots, plus gains that carry to new tasks.
Catalog record
| CREATOR | Yifei Wang, Feng Xiong, Yong Wang, Linjing Li, Xiangxiang Chu, Daniel Dajun Zeng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15709
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position Bias Mitigates Position Bias: Mitigate Position Bias Through Inter-Position Knowledge Distillation Yifei Wang, Feng Xiong, Yong Wang, Linjing Li, Xiangxiang Chu, Daniel Dajun Zeng Abstract Positional bias (PB), manifesting as non-uniform sensitivity across different contextual locations, significantly impairs long-context comprehension and processing capabilities. While prior work seeks to mitigate PB through modifying the architectures causing its emergence, significant PB still persists. To address PB effectively, we introduce Pos2Distill , a position to position knowledge distillation framework. Pos2Distill transfers the superior capabilities from advantageous positions to less favorable ones, thereby reducing the huge performance gaps. The conceptual principle is to leverage the inherent, position-induced disparity to counteract the PB itself. We identify distinct manifestations of PB under r etrieval and r easoning paradigms, thereby designing two specialized instantiations: Pos2Distill-R 1 and Pos2Distill-R 2 respectively, both grounded in this core principle. By employing the Pos2Distill approach, we achieve enhanced uniformity and significant performance gains across all contextual positions in long-context retrieval and reasoning tasks. Crucially, both specialized systems exhibit strong cross-task generalization mutually, while achieving superior performance on their respective tasks. 1 Introduction Who tied the bell could be the one to untie it. — Chinese proverb Large Language Models (LLMs) are increasingly proficient in handling long contexts, which has been unlocked by key innovations in efficient attention mechanisms Dao et al. 2022 ; Ainslie et al. 2023 and length extrapolation techniques He et al. 2024 ; Chi et al. 2023b ; Press et al. 2021 . These breakthroughs enable LLMs to tackle complex question answering over substantially larger context windows Agarwal et al. 2024 , marking a crucial step towards more capable and versatile natural language systems. Figure 1: Motivation for Pos2Distill . Marked performance decline from the 1st to 10th position in multi-document QA underscores severe PB. From an alternative viewpoint, superior responses at advantageous positions provide effective supervisory signals for less optimal positions. Nevertheless, recent studies point out a critical limitation: LLMs do not uniformly extract and utilize information across long context, consistently favoring information located at the context edges while neglecting that in the middle. This phenomenon, commonly termed the lost in the middle problem Liu et al. 2024 , underscores a pervasive and intrinsic PB inherent in long-context handling. PB poses significant obstacles in information-rich settings, such as retrieval-augmented generation, long-context reasoning Li et al. 2024a ; Kuratov et al. 2024 , and LLM-as-a-judge Wang et al. 2024b ; Li et al. 2024c . When critical information is distributed arbitrarily throughout the input, LLMs fail to identify and integrate gold content Baker et al. 2024a , culminating in unexpected model failures across various applications. To alleviate PB, prior research has delved into its underlying causes, seeking to modify key architectural components or internal representations linked to uneven contextual sensitivity Zhang et al. 2024c ; Chen et al. 2024 . Despite recent progress in narrowing the performance gap, a substantial disparity in information utilization between advantageous and disadvantaged positions still persists. Another line resorts to intensive contextual awareness training An et al. 2024 ; Zhang et al. 2024a by synthesizing training data with fine-grained information awareness Zhang et al. 2024b . However, such data-driven approaches typically incur substantial costs in both data synthesis and computational resources. Consequently, there remains a critical need for effective and efficient strategies to mitigate PB that overcome these limitations. Inspired by the proverb 1 , we contend that PB not only imposes challenges but also implicitly reveals gold signals, which can be exploited to mitigate position-induced disparity (Fig. 1 ). To this end, we introduce Pos2Distill , a novel position to position knowledge distillation framework, transferring knowledge from advantageous positions to rectify responses at unfavorable ones. Our analysis further reveals that PB exhibits distinct behavior under retrieval and reasoning paradigms. In retrieval tasks, PB predominantly manifests as token-shifting, whereas in reasoning tasks, PB interacts with Chain-of-Thought (CoT) processes Wei et al. 2022 , leading to thought-shifting, characterized by deviations in the reasoning trajectory. Accordingly, we develop two systems: Pos2Distill-R 1 and Pos2Distill-R 2 . Pos2distill-R 1 mitigates token-shifting in retrieval by incorporating Kullback-Leibler (KL) divergence loss Kullback and Leibler 1951 , providing fine-grained corrective signals. Pos2Distill-R 2 addresses thought-shifting in reasoning tasks by distilling high-quality CoT responses from advantageously positioned inputs to guide and rectify reasoning trajectories at less favorable positions. Extensive experiments demonstrate that Pos2Distill leads to more uniform and substantially improved performance both for in-context retrieval and reasoning tasks. Furthermore, data efficiency is a notable property of our method: with only 250 training instances, Pos2Distill increases the performance of Mistral-7B-v0.3 on the NQ dataset by 6.7%, as indicated in Fig. 5 . Our contributions can be summarized as follows: • We uncover distinct manifestations of PB under retrieval and reasoning paradigms, namely token-shifting and thought-shifting , offering deeper insights into the nature of PB. • We propose Pos2Distill , a novel position to position KD framework to mitigate PB. Given the different behavior of PB, we design two systems tailored to address the PB in both scenarios. • Extensive experiments and ablation studies thoroughly demonstrate the efficacy in terms of performance, data efficiency, and superior generalization. 2 Related Work Figure 2: Left: long-term decay effect property of RoPE; Right: Attention sink (causal attention matrix). Causes of Position Bias. Most LLMs adopt relative positional encodings Peysakhovich and Lerer 2023 , such as RoPE Su et al. 2024 and ALiBi Press et al. 2021 , integrating token relative distances into attention score computation. This design induces a long-range decay effect in Fig. 2 (left), whereby LLMs preferentially attend to recent tokens. Concurrently, LLMs exhibit a notable bias towards the initial tokens, which can be largely attributed to the universal phenomenon, attention sink Xiao et al. 2024 ; Gu et al. 2025 , where disproportionate attention is allocated to early tokens in Fig. 2 (right), regardless of semantic significance. In addition, causal mask enforces a unidirectional flow of information, implicitly encoding positional information Haviv et al. 2022 ; Chi et al. 2023a ; Wang et al. 2024a . The interplay of these factors collectively contributes to the emergence of PB, as further elaborated in Appendix A . Mechanistic Approaches. Current approaches predominantly focus on aforementioned underlying causes of PB. A broad spectrum of interventions has been investigated, ranging from modifications to position encodings Zhang et al. 2024c ; Chen et al. 2024 ; Lin et al. 2024 and alterations to causal masks Wang et al. 2025 , to the manipulation of internal states, including attention re-weighting Hsieh et al. 2024 ; Tan et al. 2025 and hidden state manipulation Yu et al. 2024 . Despite these efforts, the substantial performance gap across token positions remains largely unmitigated. (a) Natural PB for Retrieval. (b) Compound PB for Reasoning. Figure 3: Different Behavior of Position Bias. Training Approaches. Standard next-token prediction pretraining often yields LLMs with inadequate contextual awareness Shi et al. 2024 . A research direction therefore explores specialized datasets designed to foster fine-grained information awareness Zhang et al. 2024b , training LLMs to identify pertinent information within their full context. Nevertheless, the considerable data and computational overhead limit their scalability and practical viability. 3 Behavior of Position Bias Natural PB for Retrieval. Given