Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM

Yijun Ge, Sahel Sharifymoghaddam, Jimmy Lin · 2025

Read Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM by Yijun Ge, Sahel Sharifymoghaddam, Jimmy Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-02558.txt.

Introduction

Sahel Sharifymoghaddam, Yijun Ge, and Jimmy Lin wrote a 2025 information retrieval paper revisiting BRIGHT, a benchmark of long, reasoning-heavy search queries like those used with retrieval-augmented generation. They supply strong, reproducible baselines inside three open toolkits.

While rebuilding the lexical baseline, the authors found an under-documented detail: BM25 weighting applied to the query side, a variant called BM25Q. On long multi-sentence queries, that variant beat standard BM25 and became the strongest word-based baseline. An audit of the benchmark corpus also uncovered data quality problems, with suggested fixes. Tests on five other benchmarks show the query-weighting gain is mostly specific to BRIGHT. Blending it with standard BM25 gave the steadiest gains overall.

Catalog record

CREATORYijun Ge, Sahel Sharifymoghaddam, Jimmy Lin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-02558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM Yijun Ge, Sahel Sharifymoghaddam, Jimmy Lin Abstract The BRIGHT benchmark is a dataset consisting of reasoning-intensive queries over diverse domains. We explore retrieval results on BRIGHT using a range of retrieval techniques, including sparse, dense, and fusion methods, and establish reproducible baselines. We then apply listwise reranking with large language models (LLMs) to further investigate the impact of reranking on reasoning-intensive queries. These baselines are integrated into popular retrieval and reranking toolkits Anserini, Pyserini, and RankLLM, with two-click reproducibility that makes them easy to build upon and convenient for further development. While attempting to reproduce the results reported in the original BRIGHT paper, we find that the provided BM25 scores differ notably from those that we obtain using Anserini and Pyserini. We discover that this difference is due to BRIGHT’s implementation of BM25, which applies BM25 on the query rather than using the standard bag-of-words approach, as in Anserini, to construct query vectors. This difference has become increasingly relevant due to the rise of longer queries, with BRIGHT’s lengthy reasoning-intensive queries being a prime example, and further accentuated by the increasing usage of retrieval-augmented generation, where LLM prompts can grow to be much longer than “traditional” search engine queries. Our observation signifies that it may be time to reconsider BM25 approaches going forward in order to better accommodate emerging applications. To facilitate this, we integrate query-side BM25 into both Anserini and Pyserini. 1 Introduction The growing use of large language models (LLMs) and retrieval-augmented generation (RAG) lewis2020retrieval ; ram2023context is transforming the role of information retrieval. In RAG, LLMs use retrieved documents as context to produce more accurate and grounded responses. The BRIGHT benchmark bright was developed to evaluate retrieval systems specifically for this application, focusing on their ability to handle the complex, reasoning-intensive queries often found in user prompts. To facilitate further research with the BRIGHT benchmark, we have integrated reproducible baselines into two popular information retrieval (IR) toolkits, Anserini anserini ; anserini2 and Pyserini pyserini , for convenient first-stage retrieval. The baselines we incorporate include BM25 bm25 , a classic lexical retrieval algorithm; BGE-large-en-v1.5 bge , a representative model for dense retrieval; and SPLADE-v3 spladev3 , a representative model for learned sparse retrieval. Additionally, we have incorporated BRIGHT into RankLLM rankllm , a widely-used Python package for second-stage reranking with LLMs. This integration is particularly valuable given that LLM-based rerankers, such as those demonstrated in previous work LRL ; RankGPT ; rankzephyr ; first , have shown to provide significant improvements in retrieval effectiveness. We also discuss some of our findings about BRIGHT during this process. In our efforts to establish these baselines, we find that the BM25 results reported in the original BRIGHT paper bright differ from what we obtained initially using Anserini and Pyserini. This is because for BM25 retrieval, by default, Anserini uses bag-of-words to generate sparse query vectors, i.e., the weight of each token in the query vector is the number of times the token appears in the query, whereas the implementation used by the BRIGHT paper applies the BM25 scoring algorithm to each query token to obtain token weights. In both cases, similarity scores are the inner product between the query and document vectors. For simplicity, we refer to the former approach as “bag-of-words” and the latter as “query-side BM25”. Traditionally, BM25 is not applied to the query robertson1995Okapi ; robertson1996okapi ; bm25 , but it may be time to reconsider this as on average, query-side BM25 achieves better results than bag-of-words for BRIGHT. To reflect the more detailed, reasoning-intensive nature of user prompts for LLMs, BRIGHT queries are significantly longer than the typical keyword-based queries found in traditional information retrieval benchmarks like BEIR beir and TREC 2019-2023 Deep Learning Tracks dl19 ; dl20 ; dl21 ; dl22 ; dl23 . This shift in query style reflects how users interact with LLMs, providing more context to get better responses, as opposed to the concise keyword queries used for search engines. Our primary contribution is a set of solid and reproducible baselines for the BRIGHT benchmark, covering BM25 as well as sparse and dense retrieval techniques. These baselines are integrated into popular IR toolkits, including Anserini, Pyserini, and RankLLM, making it easy for the research community to build upon our work. We also implement and discuss the difference between bag-of-words and query-side BM25, providing insights into their effectiveness and the implications for retrieval. To facilitate future research, we have added support for query-side BM25 in Anserini and Pyserini. Finally, we share observations on BRIGHT intended to guide future improvements. 2 Background and Related Work BRIGHT is a benchmark designed to evaluate how well retrieval systems handle reasoning-intensive tasks, which remains a significant challenge for current state-of-the-art methods. The difficulty of the benchmark is reflected in the leaderboard, 1 1 1 https://brightbenchmark.github.io/ where the highest nDCG@10 is only about 0.46 as of August 31, 2025. A common way to improve effectiveness is to use LLMs, either to expand the query before retrieval or to rerank first-stage retrieval results. This pattern is evident from the BRIGHT leaderboard, where the top-performing systems, ranked by nDCG@10, all incorporate some form of LLM involvement. Notably, systems using LLM-based query expansion achieve nDCG@10 scores more than 0.1 points higher than those using the original queries brightEntry . The BRIGHT benchmark facilitates this research by providing LLM-expanded versions of its queries. Along with that, it also provides a series of first-stage retrieval results, which do poorly on their own compared to the systems with LLM involvement, but are nevertheless an important part of the pipeline for the retrieval goal bright . We reproduce two of their first-stage retrieval results, with BM25 and BGE-large-en-v1.5. BM25 is a widely used lexical matching algorithm that assigns a relevance score to documents based on several factors: the number of times a query term appears in a document, the document’s length, and the overall rarity of the term across the entire document collection. The algorithm was originally developed by combining two earlier versions, BM11 and BM15, through the introduction of a new parameter, b allowing to allocate different weights to the document length penalty term depending on the workload robertson1995Okapi . Several implementations of BM25 also account for the frequency of terms within the query itself whichBM25 ; robertson1995Okapi ; robertson1996okapi . While early versions of the algorithm included a saturation function for query term frequency (qtf), similar to how document term frequency is handled, later experiments suggested that this saturation effect was not significant bm25 . As a result, many modern implementations of BM25 treat query term frequency linearly, creating the query vector using bag-of-words. This method effectively counts multiple occurrences of a term in a query as separate instances to boost its importance. In contrast, applying the BM25 scoring function on the query side to generate query vectors has received little attention. Prior work bm25q has found that query term frequency has minimal impact on retrieval scores when most terms appear only once. However, for long queries—such as in document similarity tasks where the query is itself another document lin2007pubmed —query length normalization becomes important. This is particularly relevant in BRIGHT, where

Read the complete text