Whole-Pool Setwise Reranking with Long-Context Language Models

Hang Li, Chuting Yu · 2026

Read Whole-Pool Setwise Reranking with Long-Context Language Models by Hang Li, Chuting Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01782.txt.

Introduction

Li and co-authors seek a cheaper way to re-rank search results with chat models. Old re-rankers compare items in small groups. That needs many calls to the model in a row. Long-context models can now hold the whole pool at once. The team asks how far that fact can cut cost.

Their DualEnd method picks both best and worst items per call. It fills the ranked list from both ends at once. It ranks 100 items in 50 calls. Tests use nine open models on TREC data sets. The authors report far fewer calls than past methods. Token use and rank time drop by about half.

Catalog record

CREATORHang Li, Chuting Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01782

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Whole-Pool Setwise Reranking with Long-Context Language Models Hang Li, Chuting Yu Abstract Previous LLM-based passage re-rankers are often expensive and slow because the input context constraints require the LLM to make many dependent model calls. We study how recent long-context LLMs change this problem: when the full set of retrieved candidate passages can be shown to the model at once, ranking no longer has to be reconstructed from many overlapping local comparisons. We propose Whole-Pool Setwise re-ranking , where each call considers all currently unranked candidate passages, and introduce DualEnd , which identifies both the most and least relevant passages in one call. By filling the ranking from both ends, DualEnd ranks 100 candidates with 50 serial LLM calls, compared with 99 calls for comparable one-passage-at-a-time whole-pool methods. Experiments with nine open-weight LLMs on two passage re-ranking benchmarks, measuring effectiveness, call count, token use, runtime, and output reliability shows that long context is not merely more prompt space, but an opportunity to make LLM re-rankers both effective and efficient. 1 Introduction LLMs are increasingly being used for passage re-ranking: An LLM is instructed (typically in a zero-shot manner) to consider a pool of m candidate passages and form a top k ranking ( k\leq m ) with respect to a user query 16 ; 18 ; 10 ; 13 ; 14 ; 23 ; 12 ; 11 ; 22 ; 20 . Most LLM re-ranking work therefore optimizes ranking quality under a fixed prompting recipe. In practice, sorting-based re-rankers also pay in dependent model calls: later comparisons depend on earlier outcomes, so latency and repeated prompt processing scale with call count. Long context changes the design question from how to fit candidates into context to how much ranking information can be elicited from each context pass. We focus on a specific LLM re-ranker approach: Setwise 23 , where the LLM is instructed to select the most relevant passage from a set of candidates; the set of candidates forms a local window that is considered for comparison. This local window preference is then surfaced to a global ranking via sliding the window across the full set of candidates and employing sorting algorithms such as bubblesort or heapsort to construct the global ranking. The sliding window approach in Setwise is motivated by the limited tokens that can be included in the LLM context; for example, Flan-T5 used in the original Setwise method had a context window of 512 tokens 23 . Recent advances in LLM architecture and improved hardware affordances allow for much larger context lengths; e.g., Gemma 3 ( 5 ) has a 128k tokens context, and both Qwen 3.5 ( 15 ) and Gemini 3.1 ( 4 ) can exploit up to a 1M tokens context. Dividing up re-ranking candidates pools into sliding windows might still be required for large pools and long passages; however, with newer LLMs it is feasible to fit the entire candidate pool into a single context. In this paper, we consider what new affordances the Setwise method provides when all ranking candidates can fit into a single context window, and thus a global ranking can be constructed directly from the preferences expressed by the LLM. In particular, we show that under this setup the Setwise prompt can be modified so that at each selection step, instead of requiring the LLM to select the most relevant passage, the LLM can be instructed to select the most relevant (best) and the least relevant (worst) passages in the context. This modification allows an up to 9x speed-up in ranking time and 98% reduction in LLM calls, with a small effectiveness trade-off that is often not statistically significant. 2 Whole-Pool Setwise Re-ranking The original Setwise method, which we denoted as "Setwise-Windowed" (SW), takes a full ranking consists of N candidate passages 1 1 1 In this work, a candidate is a passage, we use candidate and passage interchangeably. for each query (called the live pool), and judges a windowed subset W of candidates at a time ( |W|<|N| ). The SW method applied an LLM comparator to small subset of passages W_{i}\subset N and then uses a sorting procedure ( 23 ) to produce the final ranking R . We consider Whole-Pool Setwise re-ranking (WP-Setwise) where the entire live pool can be judged ( |W|=|N| ). After the first judgement, ranked candidates are placed in R and the renaming unranked candidates are denoted N^{\prime} ( N^{\prime}\subset N ). Thus each LLM call sees the query and every currently unranked candidate. The ability to include all passages from the live pool into a single context gives us the opportunity to rethink the comparison that WP-Setwise does in each LLM-call. WP-Setwise-Top (WP-T) follows the original Setwise prompt which instructs the LLM to select the most relevant passage in the live pool N , places it at the next free head position of the final output ranking ( R ), and removes it from the live pool N . We extend this to instruct the LLM for the most relevant and the least relevant passages in the live pool are selected at the same time. Then, the most relavant passage is placed at the next free head position of R and the least relevant is placed in the next free tail position of R , and both passages are removed from the live pool. We call this approach WP-Setwise-DualEnd (WP-DE), because the final ranking is being filled from the two ends at the same time: from the top and from the bottom. Figure 1 illustrates the process implemented by WP-Setwise-DualEnd; the final row is the constructed output ranking R , with blue choices filling from the head and orange choices filling from the tail. The algorithm and prompts used in WP-Setwise are reported in Appendix A.1 . A third variant can be created, in which at each LLM call only the least relevant passage is selected; which we denote WP-Setwise-Bottom (WP-B). Figure 1: WP-Setwise-DualEnd fills the ranking from both ends as the live pool shrinks. Most (blue) and least (orange) relevant documents are removed jointly per LLM call; head/tail of R are filled inward. For an input ranking of N candidates, WP-T and WP-B require N-1 LLM calls to provide a full re-ranking, whereas WP-DE would reduce the number of calls to N/2 . 2 2 2 Our codebase is released under Apache 2.0 license at https://anonymous.4open.science/r/Whole-Pool-Setwise-A0E2 . Figure 2: Cost-effectiveness view of the pool-100 results. Each point is one re-ranking method, with x-axis position given by serial LLM calls per query and y-axis position given by DL19/20 mean nDCG@10. Method names are labeled directly next to their markers: red stars indicate WP methods, and black squares indicate SW methods. The x-axis is broken once after 500 calls to keep both the practical low-call region and the high-call SW-BB baseline visible. BM25 is omitted because it uses zero LLM calls. 3 Experimental Setup We evaluate on TREC Deep Learning 2019/2020 (DL19/20) ( 2 ; 3 ) , containing 43/54 judged queries, respectively. The passage collection is MS MARCO ( 1 ) containing \approx 8.8M passages. We re-rank the identical BM25 top- N pools from Pyserini ( 7 ) without adding passages outside

Read the complete text