PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models

Kaustubh D. Dhole · 2024

Read PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models by Kaustubh D. Dhole (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05339.txt.

Introduction

Kaustubh Dhole, a solo author, introduces a software plugin in a 2024 paper. Using large language models to rerank search results takes much trial and error, since prompt form, model choice, and query rewriting all change outcomes. The plugin, PyTerrier-GenRank, plugs such models into the PyTerrier search toolkit, supports pointwise and listwise prompting, and is checked through hosted services from HuggingFace and OpenAI.

Catalog record

CREATORKaustubh D. Dhole
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE6 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PyTerrier-GenRank: The PyTerrier Plugin for Reranking with Large Language Models
Kaustubh D. Dhole

Abstract
Using LLMs as rerankers requires experimenting with various hyperparameters, such as prompt formats, model choice, and reformulation strategies. We introduce PyTerrier-GenRank , a PyTerrier plugin to facilitate seamless reranking experiments with LLMs, supporting popular ranking strategies like pointwise and listwise prompting. We validate our plugin through HuggingFace and OpenAI hosted endpoints.



1
Introduction

Generative reranking methods have recently gained popularity for their ability to produce a more optimal ranking of documents by prompting large language models. These methods are also employed in Retrieval Augmented Generation
Dhole 2024
paradigms as they are effective for enhancing generation through the incorporation of retrieved information. Typically, a sparse retriever such as BM25
Robertson et al. 1992
or a dense retriever like ColBERT
Khattab and Zaharia 2020
is utilized to retrieve a small subset (ranging from 20 to 1000) from among millions or billions of text pieces. This subset is then forwarded to a second-stage generative reranker for reranking.

The increasing adoption of such reranking methods necessitates rigorous testing of various hyperparameters, including employing different prompts, evaluating performance with different model weights, and using it in conjunction with various retrieval, reranking, and reformulation
Dhole et al. 2024a
;
Dhole et al. 2024b
pipelines.

In that regard, we introduce –
PyTerrier-GenRank
– to encourage rapid experimentation of retrieval pipelines employing recently popular LLM-based reranking methods. The code includes PyTerrier
Macdonald and Tonellotto 2020
wrappers over many helper functions of the RankLLM repository
Pradeep et al. 2023a
;
Pradeep et al. 2023b
. Our plugin can be quickly installed in any Python environment and is available
at this link
1
1

1

github.com/emory-irlab/pyterrier_genrank
.

⬇

1

2

import

pyterrier

as

pt

3

from

rerank

import

LLMReRanker

4

5

bm25

=

pt
.
BatchRetrieve
(
dataset
.
get_index
(),

wmodel
=
"BM25"
)

6

7

reranker

=

LLMReRanker
(
"castorini/rank_vicuna_7b_v1"
)

8

9

pipeline

=

bm25

%

100

10

>>

pt
.
text
.
get_text
(
index
,

’text’
)

11

>>

reranker

12

13

pipeline
.
search
(
’Indian

restaurants’
)

Listing 1:
Retrieving using a listwise reranker

2
Generative Reranking

Large Language Models (LLMs) have been widely used for reranking tasks in various ways involving novel hyperparameters. Among the most popular ones have been ranking documents in a pointwise, pairwise and listwise fashion. MonoBERT
Nogueira et al. 2019
and MonoT5
Nogueira et al. 2020
are among the earliest pointwise ranking methods that were trained to emit a single scalar relevance score for a query-document pair. In contrast, pairwise approaches
Qin et al. 2024
involved prompting models with a query and two documents to compare and rank them.
RankVicuna

Pradeep et al. 2023a
,
RankZephyr

Pradeep et al. 2023b
, and
RankGPT

Sun et al. 2023
;
OpenAI 2023
have explored listwise ranking strategies by prompting their respective models with a list of documents and generating a ranked list of document IDs. These paradigms have also been investigated in the context of recommender systems
Yue et al. 2023
.

The number of hyperparameters involved in creating an optimal retriever, the choice of prompts, the number of documents, the weight of the reformulation strategy, etc., all influence the final ranking set in various ways. Besides, the increasing number of LLMs released by various organizations further necessitates rigorous testing to evaluate performance across different settings. To ensure the validity and reproducibility of results, it is imperative that researchers and developers can easily replicate experiments across these novel sets of hyperparameters.

3
Python IR Toolkits

We choose PyTerrier
Macdonald and Tonellotto 2020
due to its modularity and Python-friendly interface. It is built on the Terrier search engine and excels in modular, rapid experimentation for IR research, allowing users to easily integrate and evaluate various retrieval and re-ranking models. It has been employed in various studies to evaluate retrieval effectiveness
Dhole and Agichtein 2024b
;
Datta et al. 2024
;
Parry et al. 2024
;
Datta et al. 2022
;
Dhole and Agichtein 2024a
.

Training

Name

nDCG@10

0-shot

BM25

.480

0-shot

gpt-35-turbo
β

OpenAI 2023

.660

0-shot

gpt-4-turbo-0409
β

OpenAI 2024a

.701

0-shot

gpt-4o-mini
OpenAI 2024b

.710

0-shot

Llama-3-8B-Instruct
Meta 2024a

.572

0-shot

Llama-3.1-8B-Instruct
Meta 2024b

.594

0-shot

Llama-Spark (8B)
Arcee 2024

.612

Trained

Rank-Vicuna
β

Pradeep et al. 2023a

.672

Trained

Rank-Zephyr
β

Pradeep et al. 2023b

.711

Table 1:
Comparing reranking performance of different models for TREC-DL 2019
Craswell et al. 2020
using PyTerrier_Genrank

4
Generative Reranking Wrappers

Our plugin employs PyTerrier wrappers over reranking class functions modified from the Pyserini
Lin et al. 2021
based
RankGPT
and
RankLLM
repositories. We allow additional hyperparameters to be expressed from function calls. We provide sample code to execute HuggingFace
Wolf et al. 2020
models and OpenAI-based endpoints.

5
Sanity Check

As a sanity check, we re-evaluate some previously published rerankers like RankVicuna, RankGPT, using this framework. Those marked with superscript
β
have already been previously tested with the Pyserini framework in their respective papers. We also evaluate the zero-shot retrieval effectiveness of newly published instruction-tuned models like LLama-Spark.

We find that among the 8B zero-shot approaches, LLama-Spark is the most effective as a reranker. The complete results are shown in Table
1
.

Ethics Statement

Large Language Models are embedded within a broader socio-technical framework
Dhole 2023
;
Dhole et al. 2023

Read the complete text