Towards Reliable Testing for Multiple Information Retrieval System Comparisons
David Otero, Javier Parapar, Álvaro Barreiro · 2025
Read Towards Reliable Testing for Multiple Information Retrieval System Comparisons by David Otero, Javier Parapar, Álvaro Barreiro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03930.txt.
Description
David Otero and co-authors test how to compare many search systems at once without fooling ourselves. Significance tests judge whether one system truly beats another, but running many pairwise tests inflates false alarms. Most past work studied pairs of systems, while real retrieval trials line up several at once.
The team judged rival procedures on simulated data and on real TREC collections. The winning recipe pairs the Wilcoxon test with the Benjamini-Hochberg correction. It held false alarm rates at the promised level for typical sample sizes. It also kept the best power to spot real gaps among the options tried. The paper offers experimenters a firmer footing for crowded leaderboards.
Catalog record
| CREATOR | David Otero, Javier Parapar, Álvaro Barreiro |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03930