PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models
Kaustubh D. Dhole · 2024
Read PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models by Kaustubh D. Dhole (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05339.txt.
Introduction
Kaustubh Dhole, a solo author, introduces a software plugin in a 2024 paper. Using large language models to rerank search results takes much trial and error, since prompt form, model choice, and query rewriting all change outcomes. The plugin, PyTerrier-GenRank, plugs such models into the PyTerrier search toolkit, supports pointwise and listwise prompting, and is checked through hosted services from HuggingFace and OpenAI.
Catalog record
| CREATOR | Kaustubh D. Dhole |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 6 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05339
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models Kaustubh D. Dhole Abstract Using LLMs as rerankers requires experimenting with various hyperparameters, such as prompt formats, model choice, and reformulation strategies. We introduce PyTerrier-GenRank , a PyTerrier plugin to facilitate seamless reranking experiments with LLMs, supporting popular ranking strategies like pointwise and listwise prompting. We validate our plugin through HuggingFace and OpenAI hosted endpoints. 1 Introduction Generative reranking methods have recently gained popularity for their ability to produce a more optimal ranking of documents by prompting large language models. These methods are also employed in Retrieval Augmented Generation Dhole 2024 paradigms as they are effective for enhancing generation through the incorporation of retrieved information. Typically, a sparse retriever such as BM25 Robertson et al. 1992 or a dense retriever like ColBERT Khattab and Zaharia 2020 is utilized to retrieve a small subset (ranging from 20 to 1000) from among millions or billions of text pieces. This subset is then forwarded to a second-stage generative reranker for reranking. The increasing adoption of such reranking methods necessitates rigorous testing of various hyperparameters, including employing different prompts, evaluating performance with different model weights, and using it in conjunction with various retrieval, reranking, and reformulation Dhole et al. 2024a ; Dhole et al. 2024b pipelines. In that regard, we introduce – PyTerrier-GenRank – to encourage rapid experimentation of retrieval pipelines employing recently popular LLM-based reranking methods. The code includes PyTerrier Macdonald and Tonellotto 2020 wrappers over many helper functions of the RankLLM repository Pradeep et al. 2023a ; Pradeep et al. 2023b . Our plugin can be quickly installed in any Python environment and is available at this link 1 1 1 github.com/emory-irlab/pyterrier_genrank . ⬇ 1 2 import pyterrier as pt 3 from rerank import LLMReRanker 4 5 bm25 = pt . BatchRetrieve ( dataset . get_index (), wmodel = "BM25" ) 6 7 reranker = LLMReRanker ( "castorini/rank_vicuna_7b_v1" ) 8 9 pipeline = bm25 % 100 10 >> pt . text . get_text ( index , ’text’ ) 11 >> reranker 12 13 pipeline . search ( ’Indian restaurants’ ) Listing 1: Retrieving using a listwise reranker 2 Generative Reranking Large Language Models (LLMs) have been widely used for reranking tasks in various ways involving novel hyperparameters. Among the most popular ones have been ranking documents in a pointwise, pairwise and listwise fashion. MonoBERT Nogueira et al. 2019 and MonoT5 Nogueira et al. 2020 are among the earliest pointwise ranking methods that were trained to emit a single scalar relevance score for a query-document pair. In contrast, pairwise approaches Qin et al. 2024 involved prompting models with a query and two documents to compare and rank them. RankVicuna Pradeep et al. 2023a , RankZephyr Pradeep et al. 2023b , and RankGPT Sun et al. 2023 ; OpenAI 2023 have explored listwise ranking strategies by prompting their respective models with a list of documents and generating a ranked list of document IDs. These paradigms have also been investigated in the context of recommender systems Yue et al. 2023 . The number of hyperparameters involved in creating an optimal retriever, the choice of prompts, the number of documents, the weight of the reformulation strategy, etc., all influence the final ranking set in various ways. Besides, the increasing number of LLMs released by various organizations further necessitates rigorous testing to evaluate performance across different settings. To ensure the validity and reproducibility of results, it is imperative that researchers and developers can easily replicate experiments across these novel sets of hyperparameters. 3 Python IR Toolkits We choose PyTerrier Macdonald and Tonellotto 2020 due to its modularity and Python-friendly interface. It is built on the Terrier search engine and excels in modular, rapid experimentation for IR research, allowing users to easily integrate and evaluate various retrieval and re-ranking models. It has been employed in various studies to evaluate retrieval effectiveness Dhole and Agichtein 2024b ; Datta et al. 2024 ; Parry et al. 2024 ; Datta et al. 2022 ; Dhole and Agichtein 2024a . Training Name nDCG@10 0-shot BM25 .480 0-shot gpt-35-turbo β OpenAI 2023 .660 0-shot gpt-4-turbo-0409 β OpenAI 2024a .701 0-shot gpt-4o-mini OpenAI 2024b .710 0-shot Llama-3-8B-Instruct Meta 2024a .572 0-shot Llama-3.1-8B-Instruct Meta 2024b .594 0-shot Llama-Spark (8B) Arcee 2024 .612 Trained Rank-Vicuna β Pradeep et al. 2023a .672 Trained Rank-Zephyr β Pradeep et al. 2023b .711 Table 1: Comparing reranking performance of different models for TREC-DL 2019 Craswell et al. 2020 using PyTerrier_Genrank 4 Generative Reranking Wrappers Our plugin employs PyTerrier wrappers over reranking class functions modified from the Pyserini Lin et al. 2021 based RankGPT and RankLLM repositories. We allow additional hyperparameters to be expressed from function calls. We provide sample code to execute HuggingFace Wolf et al. 2020 models and OpenAI-based endpoints. 5 Sanity Check As a sanity check, we re-evaluate some previously published rerankers like RankVicuna, RankGPT, using this framework. Those marked with superscript β have already been previously tested with the Pyserini framework in their respective papers. We also evaluate the zero-shot retrieval effectiveness of newly published instruction-tuned models like LLama-Spark. We find that among the 8B zero-shot approaches, LLama-Spark is the most effective as a reranker. The complete results are shown in Table 1 . Ethics Statement Large Language Models are embedded within a broader socio-technical framework Dhole 2023 ; Dhole et al. 2023