Towards Reliable Testing for Multiple Information Retrieval System Comparisons

David Otero, Javier Parapar, Álvaro Barreiro · 2025

Read Towards Reliable Testing for Multiple Information Retrieval System Comparisons by David Otero, Javier Parapar, Álvaro Barreiro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03930.txt.

Description

David Otero and co-authors test how to compare many search systems at once without fooling ourselves. Significance tests judge whether one system truly beats another, but running many pairwise tests inflates false alarms. Most past work studied pairs of systems, while real retrieval trials line up several at once.

The team judged rival procedures on simulated data and on real TREC collections. The winning recipe pairs the Wilcoxon test with the Benjamini-Hochberg correction. It held false alarm rates at the promised level for typical sample sizes. It also kept the best power to spot real gaps among the options tried. The paper offers experimenters a firmer footing for crowded leaderboards.

Catalog record

CREATORDavid Otero, Javier Parapar, Álvaro Barreiro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-03930

Browse the full library · About the Tower of Babel