Active Learners as Efficient PRP Rerankers
Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero · 2026
Read Active Learners as Efficient PRP Rerankers by Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-14236.txt.
Introduction
Paschmann and five co-authors reframe pairwise ranking prompting as active learning from noisy comparisons. Pairwise Ranking Prompting elicits preference judgments from a language model, usually aggregated by sorting algorithms. Judgments are noisy, order-sensitive and intransitive, so sorting assumptions fit poorly. Truncating a full permutation to meet a call budget yields no dependable top-K. Active rankers serve as drop-in replacements improving NDCG at 10 per call under call limits. A randomized-direction oracle using one model call per pair turns position bias into zero-mean noise.
Catalog record
| CREATOR | Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35929 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.14236
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Active Learners as Efficient PRP Rerankers Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero Abstract Pairwise Ranking Prompting (PRP) elicits pairwise preference judgments from an LLM, which are then aggregated into a ranking, usually via classical sorting algorithms. However, judgments are noisy, order-sensitive, and sometimes intransitive, so sorting assumptions don’t match the setting. Because sorting aims to recover a full permutation, truncating it to meet a call budget does not produce a dependable top- K K . We thus reframe PRP reranking as active learning from noisy pairwise comparisons and show active rankers are drop-in replacements that improve NDCG@10 per call in the call-constrained regime. Our noise-robust framework also introduces a randomized-direction oracle that uses a single LLM call per pair. This approach converts systematic position bias into zero-mean noise, enabling unbiased aggregate ranking without the cost of bidirectional calls. 1 1 1 Code available at https://github.com/jerecoder/IReranker 1 Introduction LLMs are increasingly used for reranking in Retrieval-Augmented Generation (RAG): given a query and a candidate list, rerankers aggregate LLM pairwise preferences into an ordered top- K subset that strongly affects downstream answer quality ( Zhou et al. 2025 ; Zhu et al. 2023 ; Dong et al. 2024 ; Sun et al. 2025 ) . Major cloud providers now offer reranking as managed services, making call efficiency a first-class concern: LLM invocations dominate cost and latency, and the goal is a sorted prefix, not the full list. Commonly, PRP is paired with classical sorting algorithms ( Qin et al. 2024 ; Sun et al. 2023 ) : PRP supplies noisy preference judgments while sorting determines which pairs to query. This is structurally mismatched: sorting assumes transitive comparisons, while LLM judgments are stochastic and can violate transitivity. Sorting thus wastes budget polishing an unstable permutation rather than improving the top- K . LLM order effects, where swapping document presentation order can flip the judge’s choice between documents, further complicate matters ( Shi et al. 2024 ; Yin et al. 2025 ; Jeong et al. 2025 ) . Standard PRP queries both prompt directions at 2 calls per pair ( Qin et al. 2024 ; Wu et al. 2025 ) , yet preference cycles persist. We therefore frame PRP reranking as active learning from noisy pairwise comparisons, choosing adaptively which pairs to query to maximize top- K quality within a budget. This connects to the literature on best- K identification under stochastic feedback ( Mohajer et al. 2017 ; Heckel et al. 2016 ; Shah and Wainwright 2018 ; Ren et al. 2020 ; Luo et al. 2024 ) . We also evaluate a cheaper oracle: randomizing the prompt direction yields a one-call estimate that converts position bias into zero-mean noise. We study two questions: (Q1) Does active ranking outperform state-of-the-art PRP rerankers at fixed budget (NDCG@10)? (Q2) Does randomized-direction prompting improve the NDCG@10–cost trade-off beyond scheduling alone? The best-performing active scheduler in our experiments is the algorithm of Mohajer et al. 2017 , which we call Mohajer : it adaptively selects which pairs to query, concentrating comparisons near the top- K boundary. Q1 : On TREC DL2019/2020 with Flan-T5-XL, Mohajer outperforms the best sorting baseline by + 9.7 NDCG@10 at B{=}300 calls (66.1 vs. 56.4), under the same bidirectional oracle, with the advantage holding across the entire call-constrained regime ( B{=}200 – 450 ). Q2 : Randomized-direction prompting improves both strategies, but in different ways. For PRP rerankers, it raises quality at fixed budget: BubbleSort gains + 5.5 NDCG@10 at B{=}300 (56.4 \to 62.0) simply by halving the call cost per pair and covering more comparisons. For active rankers, the effect is more pronounced: comparing Mohajer under both oracles, the randomized-direction oracle raises the quality ceiling from 66.96 to 68.0 while reducing the calls needed to reach it from B{=}450 to B{=}250 , a 44% reduction. Across BEIR-style tasks, active rankers reach NDCG@10 comparable to QuickSort (Avg. 56.8 for Flan-T5-XL) with up to 7 \times fewer calls. 2 Related Work Pairwise LLM reranking. PRP elicits pairwise preferences from an LLM and aggregates them into a ranking ( Sun et al. 2023 ; Qin et al. 2024 ) , typically via sorting algorithms that assume transitivity and target an unbudgeted complete order. Order effects. LLM comparisons are direction-sensitive ( Shi et al. 2024 ; Yin et al. 2025 ; Jeong et al. 2025 ) , so PRP often queries both prompt orders, doubling cost ( Qin et al. 2024 ; Wu et al. 2025 ) . Our randomized-direction oracle makes one call per pair, producing aggregate outcomes robust to direction bias. PRP beyond sorting. PRP-Graph uses adaptive pairings ( Luo et al. 2024 ) and tournament designs structure comparisons ( Chen et al. 2024 ) . We reframe reranking as active learning from noisy feedback and evaluate active top- K identifiers as drop-in replacements for sorting ( Heckel et al. 2016 ; Shah and Wainwright 2018 ; Ren et al. 2020 ; Mohajer et al. 2017 ) , using adaptive pairings in the spirit of PRP-Graph but grounded in noise-tolerant active ranking theory. Complementary paradigms. Setwise and listwise methods ( Zhuang et al. 2024 ; Huang et al. 2025 ; Wang et al. 2025 ) reduce cost by processing multiple documents per call, changing the prompting primitive itself; pairwise and listwise calls differ in token cost, context length, and bias, making raw call counts incommensurable across paradigms. Our goal is to improve scheduling within pairwise PRP, which remains widely deployed for its fine-grained signal and constrained-output reliability ( Qin et al. 2024 ) ; the two directions are complementary. Number of LLM calls (budget) Oracle Ranker 100 150 200 250 300 350 400 450 500 Bidirectional BubbleSort 49.27 49.27 56.43 56.43 56.42 56.98 60.25 60.30 60.51 HeapSort 6.81 6.13 6.13 9.28 23.04 41.84 54.29 62.81 68.21 QuickSort 55.93 55.89 55.87 56.20 56.20 56.20 56.40 56.59 56.68 PAC + Bubble 49.27 49.27 49.27 49.27 57.52 60.59 60.61 † 60.61 60.61 Mohajer + Bubble 30.12 30.12 62.34 64.80 66.09 66.28 66.83 67.02 † 67.02 Mohajer 30.12 30.12 62.34 64.80 66.09 66.28 66.81 66.96 † 66.96 Randomized BubbleSort 55.90 \pm 0.28 56.10 \pm 0.21 59.82 \pm 0.20 59.68 \pm 0.21 61.95 \pm 0.21 62.03 \pm 0.18 64.04 \pm 0.19 64.00 \pm 0.25 65.42 \pm 0.30 HeapSort 6.58 \pm 0.14 16.46 \pm 0.43 50.17 \pm 0.24 65.80 \pm 0.16 68.50 \pm 0.39 68.41 \pm 0.30 68.34 \pm 0.11 68.53 \pm 0.21 68.71 \pm 0.21 QuickSort 54.49 \pm 0.27 54.71 \pm 0.23 55.26 \pm 0.24 56.20 \pm 0.47 57.56 \pm 0.19 58.95 \pm 0.34 59.81 \pm 0.41 61.87 \pm 0.36 63.76 \pm 0.38 PAC + Bubble 49.27 \pm 0.00 57.02 \pm 0.21 60.01 \pm 0.21 † 60.01 \pm 0.21 60.01 \pm 0.20 60.01 \pm 0.21 60.01 \pm 0.20 60.01 \pm 0.21 60.01 \pm 0.21 Mohajer + Bubble 61.36 \pm 0.31 65.84 \pm 0.33 67.66 \pm 0.35 67.59 \pm 0.34 68.14 \pm 0.26 68.25 \pm 0.27 68.08 \pm 0.12 † 68.08 \pm 0.12 68.08 \pm 0.12 Mohajer 61.36 \pm 0.31 65.84 \pm 0.33 67.66 \pm 0.35 68.00 \pm 0.19 † 68.00 \pm 0.19 68.00 \pm 0.19 68.00 \pm 0.19 68.00 \pm 0.19 68.00 \pm 0.19 Table 1: Average NDCG@10 (%) on TREC DL 2019 and DL 2020 with Flan-T5-XL