Beyond Correlations: A Downstream Evaluation Framework for Query Performance Prediction

Payel Santra, Partha Basuchowdhuri, Debasis Ganguly · 2026

Read Beyond Correlations: A Downstream Evaluation Framework for Query Performance Prediction by Payel Santra, Partha Basuchowdhuri, Debasis Ganguly (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17339.txt.

Introduction

Payel Santra, Partha Basuchowdhuri, and Debasis Ganguly question how search quality tools are judged. Query performance prediction, or QPP, guesses how well a search worked. The usual test checks set-level correlation. It compares guessed quality ranks with true ranks across a query set. The authors say that test is too coarse.

It does not show how QPP works for one query at a time. It also does not link scores to real downstream use. Their framework judges QPP by what it helps users do next. That shift can change which methods look best. Search teams gain a test that tracks real value, not just neat rank match.

Catalog record

CREATORPayel Santra, Partha Basuchowdhuri, Debasis Ganguly
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Correlations: A Downstream Evaluation Framework for Query Performance Prediction Payel Santra, Partha Basuchowdhuri, Debasis Ganguly Abstract The standard practice of query performance prediction (QPP) evaluation is to measure a set-level correlation between the estimated retrieval qualities and the true ones. However, neither this correlation-based evaluation measure quantifies QPP effectiveness at the level of individual queries, nor does this connect to a downstream application, meaning that QPP methods yielding high correlation values may not find a practical application in query-specific decisions in an IR pipeline. In this paper, we propose a downstream-focussed evaluation framework where a distribution of QPP estimates across a list of top-documents retrieved with several rankers is used as priors for IR fusion. While on the one hand, a distribution of these estimates closely matching that of the true retrieval qualities indicates the quality of the predictor, their usage as priors on the other hand indicates a predictor’s ability to make informed choices in an IR pipeline. Our experiments firstly establish the importance of QPP estimates in weighted IR fusion, yielding substantial improvements of over 4.5% over unweighted CombSUM and RRF fusion strategies, and secondly, reveal new insights that the downstream effectiveness of QPP does not correlate well with the standard correlation-based QPP evaluation. 1 Introduction Query Performance Prediction (QPP) methods aim to estimate the retrieval effectiveness of a ranked list without access to relevance judgments [ 2 , 19 , 22 , 21 ] . Unsupervised QPP approaches mostly leverage the retrieval score distribution of the top documents obtained with the original query [ 20 , 19 , 21 ] or its variants [ 7 , 27 ] , along with other characteristic features, such as term informativeness [ 28 ] , co-occurrence [ 22 ] and semantics [ 8 ] . However, supervised approaches rely on the query content and retrieved documents to predict the retrieval quality [ 1 , 2 , 12 ] . In its standard form, the objective of the QPP task is to effectively distinguish easy queries from difficult ones for a given retrieval model. Accordingly, its evaluation typically focuses on measuring the correlation between the true retrieval effectiveness (e.g., AP or nDCG) and the predicted QPP scores [ 11 ] . However, existing QPP models are largely agnostic to both the underlying information retrieval (IR) model and the evaluation metric. Consequently, the observed correlations are often highly sensitive to these factors, i,e., observations are mostly not consistent across a range of different ranking models and target IR metrics [ 11 ] . Another major limitation of the correlation-based QPP evaluation paradigm is its dependence on a set of queries, in contrast to ad-hoc IR evaluation, which can be performed independently for each query — specifically, by assessing whether the retrieved documents satisfy that query’s information need. Consequently, QPP evaluation is inherently conditioned on the query set, where the performance of one query is interpreted relative to that of others, unlike per-query measures such as AP or nDCG that evaluate retrieval quality in isolation. Beyond the lack of robustness and per-query independence, the most critical shortcoming of existing QPP evaluation methodologies is their misalignment with the downstream utility of QPP estimates—the very motivation behind the task. To address this, we propose a downstream-aware QPP evaluation framework . Given an input query and the corresponding rank list, our QPP evaluation approach first employs a QPP model to predict the expected retrieval quality. This formulation enables per-query evaluation of a QPP model by predicting a relative preference of rankers for each query, while also facilitating a natural downstream application—using the predicted scores as weights in an information retrieval fusion framework. If the predicted likelihoods closely approximate the true retrieval performance, incorporating them as priors in the fusion process should enhance overall retrieval effectiveness. Consequently, a QPP model can be regarded as more effective than another if its estimates, when employed as priors in an IR fusion algorithm, yields more effective retrieval quality than its competitor. In this way, we not only contribute a downstream-oriented perspective on QPP evaluation but also demonstrate that ad hoc retrieval performance can be enhanced through QPP-based fusion. The source code for the proposed QPP evaluation framework and the QPP-based IR fusion is made available for research purposes 1 1 1 https://github.com/payelsantra/QPP-Fusion . 2 Downstream Evaluation of QPP Models on IR Fusion 2.1 QPP across multiple rankers Standard QPP is formalized as a function of the form \phi(q,L_{\theta}(q))\mapsto\mathbb{R} , which predicts the retrieval quality for a query q and its top documents L_{\theta}(q) obtained from a single ranker \theta . Instead of using these estimates to distinguish between multiple queries, we rather employ a QPP model \phi to obtain predicted retrieval qualities across several IR models, i.e., for a set of available rankers \Theta=\{\theta_{1},\ldots,\theta_{m}\} , a QPP model outputs a distribution of estimated retrieval qualities across the rankers. Formally, \phi(q,\mathcal{L}_{\Theta}(q))\mapsto\mathbb{R}^{m},\text{where}\,\mathcal{L}_{\Theta}(q)=\cup_{\theta\in\Theta}L_{\theta}(q), (1) 2.2 QPP for Weighted Fusion An IR fusion model combines the top-retrieved document lists obtained from multiple rankers into a single ranked list by aggregating the scores or ranks of each document across these lists [ 10 , 3 ] . Formally, \theta_{f}(q,d;\Theta)=\sum_{\theta\in\Theta}P(L_{\theta}(q)|d)\,\sigma(q,d;\theta), (2) where \theta_{f}:q\times d\times\Theta\mapsto\mathbb{R} denotes a generic weighted fusion scoring function, P(L_{\theta}(q)|d) represents the prior likelihood of selecting a particular document list conditioned on d , and \sigma(q,d;\theta)\in\mathbb{R} denotes the similarity scoring function of the IR model \theta . A uniform prior, P(L_{\theta}(q)|d)=1 , corresponds to the unweighted fusion method like, CombSUM [ 10 ] , while setting P(L_{\theta}(q)|d)=\sum_{\theta}\mathbb{I}(d\in L_{\theta}(q))/m (where m=|\Theta| ) yields CombMNZ [ 10 ] . These priors, P(L_{\theta}(q)|d) , can also be optimized using a set of training queries with known relevance assessments [ 15 ] . Unlike existing fusion approaches that employ uniform or maximum-likelihood based priors P(L_{\theta}(q)|d) as weights for each ranked list, we propose to use QPP estimates to determine these weights. More specifically, Equation 2 becomes \theta_{f}(q,d;\Theta,\Phi)=\sum_{\theta\in\Theta}\phi(q,L_{\theta}(q))\,\sigma(q,d;\theta) (3) where \phi(q,L_{\theta}(q)) denotes the QPP estimate for the ranked list L_{\theta}(q) so that the function \theta_{f} of Equation 2 now additionally also depends on the estimated QPP distribution: \Phi=(\phi_{1},\ldots,\phi_{m}) . The underlying intuition is that a more accurate distribution of QPP estimates across the ranked lists—one that more closely reflects their true retrieval quality—should lead to a more effective fusion outcome. More precisely, a QPP model \phi is considered to be more effective than another \phi^{\prime} if for a query q the QPP weights obtained with \Phi yield a better retrieval results, as measured with an IR metric \mathcal{P} (e.g., AP, nDCG etc.), than those with \Phi^{\prime} for q . This is clearly different from the standard downstream-agnostic based QPP evaluation mechanism, which considers a QPP model \phi to be more effective than \phi^{\prime} if the estimates of \phi over a set of queries correlate better with a target IR metric than that of \phi^{\prime} s. 3 Experimental Setup Research Questions . We conduct our experiments on the TREC DL 2019 [ 4 ] and 2020 [ 5 ] datasets to explore the following RQs. RQ1 : Does the usage of QPP estimates as prior weights in IR fusion improve the retrieval effectiveness of over non-weighted fusion methods? RQ2 : How correlated is the downstream-aware QPP evaluation with the

Read the complete text