When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking

Orian Dabod, Amir DN Cohen, Gabriel Stanovsky · 2026

Read When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking by Orian Dabod, Amir DN Cohen, Gabriel Stanovsky (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31087.txt.

Introduction

Dabod, Cohen, and Stanovsky question a common habit in few-shot learning. Systems often retrieve example prompts, then rerank them with extra model calls. The habit assumes reranking always helps. It also costs a lot: token use can rise more than tenfold. The authors show the expensive step can actually make answers worse, above all on easy inputs where the first choice was already good.

Their fix is Training-Free Gated Reranking. The model reranks only when it looks uncertain. Uncertainty is read from the perplexity of its first generation. A threshold decides which inputs pass the gate. Tests covered 8 language models, 7 language understanding datasets, and 9 translation setups. The gated method cut compute costs by 15 to 80 percent. Average performance rose by up to 2 percent at the same time.

Catalog record

CREATOROrian Dabod, Amir DN Cohen, Gabriel Stanovsky
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31087

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Reranking Hurts: Uncertainty-Based Gating for Few-Shot Reranking Orian Dabod, Amir DN Cohen, Gabriel Stanovsky Abstract Few-shot selection typically assumes that reranking retrieved examples always improves performance. We challenge this view by identifying that the expensive reranking step can in fact degrade performance. Instead, we propose Training-Free Gated Reranking , which decides whether to rerank the few-shot examples based on the model’s uncertainty. Extensive experiments across 8 LLMs, covering 7 NLU datasets and 9 MT domain-language combinations, demonstrate that our approach reduces computational costs by 15%-80% while improving average performance by up to 2%. These findings indicate that higher computational cost does not guarantee better performance, and that reranking is most beneficial when targeted at high-uncertainty instances. 1 Introduction Recent work has found that adaptive few-shot selection can improve the performance of LLMs on in-context learning tasks ( 1 ; 2 ; 7 ) . In particular, several works adopt a “retrieve-then-rerank” approach, fetching a broad pool of candidates and using trained cross-encoders 6 ; 19 ; 12 or training-free scorers to select demonstrations for a specific inference sample 22 ; 9 . However, reranking with repeated LLM calls increases computational costs, total token consumption can increase by 13.4 \times for MT and by 29.5 \times for NLU. 1 1 1 Calculated per task, as the total sum of tokens across all models and datasets. 1(a) 0 10 20 30 40 50 60 70 80 90 100 -6 % -4 % -2 % 0 % 2 % Relative BLEU Change (%) (a) Machine Translation (MT) Tasks 0 10 20 30 40 50 60 70 80 90 100 -2 % 0 % 2 % Token Saving (%) Relative Performance Change (%) (b) Natural Language Understanding (NLU) Figure 1: Relative performance impact versus computational token savings achieved via selective reranking for Machine Translation (MT, measured in BLEU) and Natural Language Understanding (NLU) tasks. The majority of models (3B–70B) exhibit performance gains, though the optimal savings window shifts from 15–50% for MT to roughly 50–80% for most NLU models. To address these concerns, we propose Training-Free Gated Reranking , a simple yet effective approach, which uses the perplexity of the LLM’s initial generation to rerank examples only when the model is uncertain. Rather than reranking every instance, we apply the reranking step exclusively when the input’s perplexity falls above a predefined threshold. Our approach improves efficiency while matching or slightly surpassing performance across various models and datasets. Figure 1 shows that tuning the perplexity threshold for reranking yields computational savings of 15%-80% while maintaining or surpassing baseline performance. Our manual analysis confirms that reranking provides clear gains for difficult instances, but can degrade performance when applied to easy instances. Overall, our approach presents a convenient tradeoff between resources and performance across 8 models ranging from 3B to 70B parameters, covering 9 combinations of language pairs and specialized domains, as well as 7 NLU datasets. 2 2 2 Code and experimental data are provided in the supplementary material for review, and will be made publicly available upon acceptance. 2 Perplexity-Based Reranking of Few-Shot Examples We propose a simple approach that adaptively allocates computational resources to choose few-shot examples based on the model’s intrinsic uncertainty. Formal definitions. Given an LLM M , an input x and a large pool of candidate demonstrations \mathcal{C}=((x_{1},y_{1}),\ldots,(x_{n},y_{n})) , each consisting of an input and a corresponding gold label, ranked according to their relevance to x , we are looking for a function: T(M,x,\mathcal{C})\mapsto\{c_{1},\ldots,c_{k}|c_{i}\in\mathcal{C}\} (1) That is, a selection of k demonstration samples from the pool according to x . Method. We start by generating a prediction without reranking \hat{y}=M(x,c_{1},\ldots,c_{k}) , and use it to compute a normalized uncertainty score U\in[0,1] . This score is derived from the inverse perplexity of the generation: \text{U}(M,x,\mathcal{C})=1-\text{PPL}_{M}(\hat{y}\mid x,c_{1},\ldots,c_{k})^{-1} (2) Where, and PPL computes the conditional perplexity of \hat{y} on x and the first k examples. U is in the range [0,1] , where larger values indicate that the model is less certain in its own prediction. We then define our demonstration selection function T based on a predefined uncertainty threshold \tau : T(M,x,\mathcal{C})=\begin{cases}((x_{1},y_{1}),\ldots(x_{k},y_{k}))&U\leq\tau,\\ \operatorname{rerank}(M,x,\mathcal{C})&U>\tau\end{cases} (3) where rerank is defined as the conditional entropy rerank 9 : \operatorname{rerank}(M,x,\mathcal{C})=\operatorname*{Top-k}_{(x_{i},y_{i})\in\mathcal{C}}\Big(-\text{PPL}_{M}(x\mid(x_{i},y_{i}))\Big) (4) In practice, we calibrate the threshold \tau to maximize performance on a development set. During this calibration phase, we apply a moving average with a window size of 5 to smooth local variance. Global (All) Small Models Bigger Models Method Perf. \uparrow Save. % Perf. \uparrow Save. % Perf. \uparrow Save. % Machine Translation (BLEU / COMET) No Reranking 37.09/81.05 100.0 33.37/79.46 100.0 38.33/81.57 100.0 Full Reranking 38.32/81.37 0.0 34.77/79.89 0.0 39.51/81.86 0.0 Gating (dev-set calibrated) 38.42 † / 81.39 20.86 34.95 † / 79.94 20.52 39.58 † / 81.88 20.98 Gating (test-set calibrated) 38.69 † / 81.43 17.42 35.19 † / 79.98 19.22 39.85 † / 81.92 16.82 Natural Language Understanding (Accuracy) No Reranking 79.88 100.0 77.16 100.0 80.78 100.0 Full Reranking 80.73 0.0 78.45 0.0 81.50 0.0 Gating (dev-set calibrated) 80.81 54.3 78.60 50.2 81.55 55.6 Gating (test-set calibrated) 81.41 † 47.16 79.30 † 46.69 82.11 † 47.32 Table 1: Performance summary across tasks and model sizes. Performance (Perf.) shows BLEU/COMET for MT and Accuracy for NLU. Bold indicates row maxima; underlined values exceed Full Reranking performance. Gating (dev-set calibrated) uses dev-set \tau , while Gating (test-set calibrated) uses test-set \tau . † denotes a statistically significant improvement over No Reranking ( p<0.05 ). Exact p -values are provided for transparency. 3 Evaluation 3.1 Experimental Setup We list below key experimental details, see more details in the Appendix. Tasks and models. We evaluate our approach across 8 LLMs spanning 3B–70B parameters, compared to a full reranking baseline 9 , assessing performance on both natural language understanding (NLU) and machine translation (MT) tasks. Specifically, we adopt the NLU benchmarks used by 9 : SST-2, SST-5 13 , CR, AgNews, Subj 16 , MNLI 21 , and QNLI 16 . For MT, we use three domain specific corpora 5 : EMEA (medical) 15 , JRC-Acquis (legal) 14 , and KDE (technical) 15 . We consider both translation directions between English and Spanish, Portuguese, and German, yielding six directions per domain. We sample 1,000 parallel sentence pairs for each domain-direction combination, partitioned into 200 development examples for threshold tuning and 800 test examples for evaluation. All results are averaged over 20 random splits for statistical robustness. Baselines. Baselines include retrieval-based k -shot selection with BM25 11 and dense retrieval using e5-base-v2 or multilingual-e5-base depending on the language 17 ; 18 , and always-rerank ( \tau=0 , reranking for every input at O(N) ). We also compare gating signals to simple input proxies (e.g., source length, source entropy) and an uncertainty proxy (token-level logit gap - TARG) 20 . Hyperparameters and metrics. We fix k=5 demonstrations and a candidate pool of N=100 , report MT quality with BLEU 8 and COMET (wmt22-comet-da) 10 , and NLU quality with accuracy. Following scaling laws for inference compute 4 and standard open-weight serving metrics 3 , we evaluate efficiency using the unweighted sum of input and output tokens across all processing stages: (a) initial draft generation, (b) second-pass generation after reranking for triggered cases, and (c) all conditional perplexity computations

Read the complete text