Re-Rankers as Relevance Judges Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (

Chuan Meng, Jiqun Liu, Mohammad Aliannejadi, Fengran Mo, Jeff Dalton, Maarten de Rijke · 2026

Read Re-Rankers as Relevance Judges Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval ( by Chuan Meng, Jiqun Liu, Mohammad Aliannejadi, Fengran Mo, Jeff Dalton, Maarten de Rijke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04455.txt.

Description

Chuan Meng and five co-authors ask whether the tools that rank search results can also judge them, in a 2026 arXiv paper. Relevance judgments — human-style labels of whether a passage answers a query — are increasingly predicted by large language models at great expense. Yet re-rankers, built to order results by relevance, are solving a near-identical problem, and reusing them could save duplicated effort.

The team adapts eight re-rankers from three families two ways: reading their generated true-or-false tokens as verdicts, or thresholding their scores into labels. On five years of TREC deep learning track data, the repurposed judges beat a leading language-model judge in around half the cases. They also caught a bias worth knowing: re-ranker judges favour results from their own and related re-rankers, a self-preference that any evaluation built on them must correct for.

Catalog record

CREATORChuan Meng, Jiqun Liu, Mohammad Aliannejadi, Fengran Mo, Jeff Dalton, Maarten de Rijke
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-04455

Browse the full library · About the Tower of Babel