Modeling Ranking Properties with In-Context Learning
Nilanjan SinhababuCentre for Computationaland Data SciencesIIT Kharagpur, India [email protected], Andrew ParrySchool of Computing ScienceUniversity of GlasgowUnited Kingdom [email protected] Debasis GangulySchool of Computing ScienceUniversity of GlasgowUnited Kingdom Debasi · 2025
Read Modeling Ranking Properties with In-Context Learning by Nilanjan SinhababuCentre for Computationaland Data SciencesIIT Kharagpur, India [email protected], Andrew ParrySchool of Computing ScienceUniversity of GlasgowUnited Kingdom [email protected] Debasis GangulySchool of Computing ScienceUniversity of GlasgowUnited Kingdom Debasi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17736.txt.
Introduction
Nilanjan Sinhababu and co-authors steer search rankings with examples rather than fresh training, in a paper from two thousand twenty-five. Real search must balance pure relevance with goals like diversity and fairness. Those goals hinge on how results interact, and past fixes needed new training for each task.
Their method shows a language-model ranker a few sample rankings that display the wanted trade-offs. The samples come from past queries much like the current one. Gains reached about twenty percent for fairness and over twenty percent for topic diversity. Relevance held steady across the test sets.
Catalog record
| CREATOR | Nilanjan SinhababuCentre for Computationaland Data SciencesIIT Kharagpur, India [email protected], Andrew ParrySchool of Computing ScienceUniversity of GlasgowUnited Kingdom [email protected] Debasis GangulySchool of Computing ScienceUniversity of GlasgowUnited Kingdom Debasi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17736
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modeling Ranking Properties with In-Context Learning Nilanjan SinhababuCentre for Computationaland Data SciencesIIT Kharagpur, India [email protected], Andrew ParrySchool of Computing ScienceUniversity of GlasgowUnited Kingdom [email protected] Debasis GangulySchool of Computing ScienceUniversity of GlasgowUnited Kingdom [email protected] Pabitra MitraDepartment of ComputerScience and EngineeringIIT Kharagpur, India [email protected] Abstract While standard IR models are mainly designed to optimize relevance, real-world search often needs to balance additional objectives such as diversity and fairness. These objectives depend on inter-document interactions and are commonly addressed using post-hoc heuristics or supervised learning methods, which require task-specific training for each ranking scenario and dataset. In this work, we propose an in-context learning (ICL) approach that eliminates the need for such training. Instead, our method relies on a small number of example rankings that demonstrate the desired trade-offs between objectives for past queries similar to the current input. We evaluate our approach on four IR test collections to investigate multiple auxiliary objectives: group fairness (TREC Fairness), polarity diversity (Touché), and topical diversity (TREC Deep Learning 2019/2020). We empirically validate that our method enables control over ranking behavior through demonstration engineering, allowing nuanced behavioral adjustments without explicit optimization. * * footnotetext: These authors contributed equally to this work. 1 Introduction Modern transformer-based language models are effective for ad-hoc ranking tasks Karpukhin et al. 2020 ; Pradeep et al. 2023 . By learning notions of relevance from sufficient training data, these approaches often outperform unsupervised rankers ( Karpukhin et al. 2020 ; Formal et al. 2021 ) . However, beyond the main objective of providing relevant content to a user, an IR system may have other auxiliary objectives, such as maximizing exposure fairness or topical diversity of documents Carbonell and Goldstein 2017 . Different from relevance, which is an individual property of a document itself, these additional objectives, such as diversity Clarke et al. 2008 or fair representation Craswell et al. 2008 , are instead properties of a top-retrieved list of documents, requiring effective modeling of inter-document interactions. Figure 1: Proposed ICL method for reranking a set of top-retrieved documents. An example constitutes a localized query along with its top-retrieved arranged to satisfy a desired ranking property, such as relevance, fairness, diversity, etc. Prior work on modeling ranking properties, such as diversity Carbonell and Goldstein 2017 involved interventions often in the form of ad hoc heuristic-based transformations of rankings Agrawal et al. 2009 . More recently, supervised approaches have been applied for learning neural interventions MacAvaney et al. 2021 , and incorporating inter-document interactions Sun et al. 2023 . A limitation of these supervised approaches is that they need to be trained on an adequate number of labeled examples for each different learning objective Schlatt et al. 2024 . To alleviate this limitation, we explore the use of instruction-tuned generative language models (LLMs) for this task – the advantage being these models can potentially act as universal ranking controllers adjusting their behavior through prompt examples alone Brown et al. 2020 . While LLMs have been shown to be effective for zero-shot ranking tasks facilitated by interpreting natural language instructions Sun et al. 2023 ; Pradeep et al. 2023 , modeling listwise objectives is more challenging because of: firstly, the difficulty of expressing them in natural language, and secondly, optimizing a prompt instruction for each different auxiliary ranking objective. Such “prompt engineering” is brittle Ishibashi et al. 2023 ; Habba et al. 2025 , requiring iterative refinement as objectives evolve. In contrast, we propose to use in-context learning (ICL) Wei et al. 2022 ; Lu et al. 2024 , where static instructions are paired with in-context examples of a desired ranking behaviour specific to a task. For instance, a single demonstration interleaving pro and con arguments could implicitly teach an LLM to diversify viewpoints, even with a task-agnostic instruction like “order by relevance”. This approach circumvents the need for task-specific prompts while accommodating composite or dynamic objectives Sinhababu et al. 2024 . As shown in Figure 1 , our method reranks candidate documents using LLMs conditioned on: (1) localized (on-topic) query examples Sinhababu et al. 2024 from a large query log without requiring relevance assessments Nguyen et al. 2016 ; Reimer et al. 2023 and (2) encoding ranking properties via list-wise demonstrations , e.g., diversity, fairness etc. Our main contributions are as follows: • A ICL-based approach that is shown to control desirable ranking properties without any supervised list-wise training. We establish that demonstrations can change behaviour in a causal manner. Crucially, ablations confirm demonstrations as the causal factor: Inverting examples significantly degrades performance, affirming their role in adapting LLM behavior. These results provide evidence for demonstration-based model adaptation as a generalizable paradigm for dynamic ranking control. • Empirical validation of significantly improving several auxiliary objectives without sacrificing relevance. Experiments on TREC DL (diversity) Nguyen et al. 2016 ; Craswell et al. 2020b and TREC Fairness 2022/Touché (fairness) Ekstrand et al. 2022 ; Bondarenko et al. 2020 validate our approach. Unlike prior work that sacrifices relevance for auxiliary objectives Zehlike and Castillo 2020 , our method maintains relevance while significantly improving diversity and fairness. We provide our source code to facilitate future research and the reproducibility of our work 1 1 1 https://anonymous.4open.science/r/GPT_ranker-7099 . 2 Related Work Ranking Models. Traditional term-weighting ranking models Robertson et al. 1994 relied on exact lexical matches – a constraint later alleviated by neural approaches that leverage contextualized language representations for semantic soft matching ( Karpukhin et al. 2020 ; Formal et al. 2021 ) . Transformer-based architectures, such as cross-encoders (which jointly process query-document pairs) and bi-encoders (which map queries and documents to separate embeddings), emerged as dominant paradigms ( Khattab and Zaharia 2020 ; Schlatt et al. 2024 ) . However, these models typically require task-specific fine-tuning via backpropagation to accommodate new objectives beyond generic relevance, limiting their adaptability. Our work diverges by eliminating gradient-based updates entirely, like other prior zero-shot neural ranking approaches ( Li et al. 2023b ; Sinhababu et al. 2024 ) , enabling a flexible framework where rankings can dynamically satisfy diverse user or system-defined criteria, such as fairness or diversity, without requiring retraining. Fairness. Prior work has emphasized the importance of balancing relevance with equitable exposure of document groups defined by attributes such as demographic origin, political stance, or gender ( Zehlike et al. 2017 ; Morik et al. 2020 ) . Biased exposure in rankings risks perpetuating systemic inequities, such as amplifying majority viewpoints while suppressing marginalized perspectives ( Craswell et al. 2008 ; Ekstrand et al. 2019 ) . Post-hoc fairness methods, including re-ranking algorithms ( Morik et al. 2020 ; Biega et al. 2018 ) and fairness-aware loss functions ( Singh and Joachims 2018 ) , explicitly redistribute exposure across groups but often degrade relevance ( Pleiss et al. 2017 ; Corbett-Davies et al. 2017 ) . Model-based approaches face challenges in defining target exposure distributions and incorporating them as an objective into a ranking loss ( Heuss et al. 2022 ; Jänich et al. 2024 ) . Furthermore, supervised methods struggle with biased training data ( Zehlike and Castillo 2020 ) and dependence on sensitive group labels, which are often incomplete or unavailable ( Zehlike et al. 2017 ; Jänich et al. 2024 ) . These limitations underscore the