Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808540 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results
Julián Urbano · 2026
Read Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808540 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results by Julián Urbano (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25349.txt.
Introduction
Urbano argues the Wilcoxon signed-rank test, oft sold as a safe swap for the t-test, rests on a false tale. Texts paint Wilcoxon as the right swap since scores are not normal. A close look shows gaps and slips in test rules, sowing IR confusion. Wilcoxon has been misused for decades, giving false safety from a nonexistent threat, while adding a flaw that ensures breakdown. Text review and TREC data show Wilcoxon loses grip on its Type I error rate in IR work. Its use is not justified.
Catalog record
| CREATOR | Julián Urbano |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56783 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.25349
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stop Using the Wilcoxon Test: Myth, Misconception and Misuse in IR Research Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3808540 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Evaluation of retrieval results Julián Urbano Abstract . In benchmarking of Information Retrieval systems, the Wilcoxon signed-rank test is often treated as a safer alternative to the t t -test. This belief is fueled by textbooks and recommendations that portray Wilcoxon as the proper non-parametric alternative because metric scores are not normally distributed. We argue that this narrative is misleading and harmful. A careful review of Statistics textbooks reveals inconsistencies and omissions in how the assumptions underlying these tests are presented, fostering confusion that has propagated into IR research. As a result, Wilcoxon has been routinely misapplied for decades, creating a false sense of safety against a threat that was never there to begin with, while introducing another one so severe that it virtually guarantees the test will break down and mislead researchers. Through a combination of systematic literature review, analysis and empirical demonstrations with TREC data, we show how and why the Wilcoxon test easily loses control of its Type I error rate in IR settings. We conclude that the continued use of Wilcoxon in IR evaluation is unjustified and that abandoning it would improve the methodological soundness of our field. † † cc-license: by 1. Introduction In Information Retrieval (IR) evaluation, system effectiveness is usually measured over the fixed set of topics in a test collection or dataset. Because topic difficulty varies, observed differences between two systems, say X and Y , inevitably reflect both genuine performance gaps and random variation due to topic sampling. Statistical testing is typically used to separate signal from noise: to ask whether the apparent improvement of X over Y is more than a coincidence due to the selection of topics in the test collection. Formally, for each topic i=1,\dots,n we record scores X_{i} and Y_{i} according to some metric like NDCG, and the null hypothesis of interest is that the systems have equal mean effectiveness in the population of topics, H_{0}:\mu_{X}=\mu_{Y} . Since both systems are tested on the same topics, this is called a paired two-sample problem , where dependence across topics (difficulty) is accounted for. This leaves us with two classical procedures that dominate the Statistics literature and IR practice, usually described as follows: (1) Paired Student t -test: a parametric test assuming that scores follow a normal distribution. (2) Wilcoxon signed-rank test: a non-parametric alternative when the normality assumption is violated. It is worth noting that once we define the per-topic differences D_{i}=X_{i}-Y_{i} , this problem is equivalent to testing the one-sample hypothesis H_{0}:\mu_{D}=0 . As we will show in Section 2.3 , this alternative view simplifies definitions and discussion. Over the decades, IR research has repeatedly asked which test is “best” for system benchmarking. A first wave of papers discussed mostly theoretical arguments ( 76 ; 34 ; 81 ) . A second wave took an empirical angle, using resampling and random topic set splits ( 59 ; 84 ; 77 ; 58 ; 54 ; 64 ; 21 ; 65 ; 10 ; 73 ) . A third wave is represented by simulation studies that recently calculated actual Type I and II errors for IR-like data, 1 1 1 As a reminder, a Type I error occurs when the null hypothesis is true but it is rejected (claiming a difference when there is none, i.e., a false positive), and a Type II error occurs when the null is not true and yet it is not rejected (missing a difference when there is one, i.e., a false negative). but reached opposite conclusions: ( 74 ; 72 ; 71 ) recommend the t -test and discourage the Wilcoxon test, while ( 49 ; 48 ) actually recommend the latter. In practice, surveys of venues like SIGIR and ECIR show that about 65% of papers use the t -test and 25% Wilcoxon, with other methods being a minority ( 55 ; 12 ) . In this paper, we argue that the Wilcoxon test should no longer be used in IR evaluation, not because it is sub-optimal, but because it is actively harmful . Unlike earlier empirical or simulation-based work, we take a different route and present a systematic review of 25 Statistics textbooks. This reveals the root of a problem: the very sources we rely upon to shape our collective understanding perpetuate a confusing and misleading dichotomy between parametric and non-parametric methods. We frame the issue in terms of myth , misconception , and misuse . The myth is that non-parametric methods are inherently safer when parametric assumptions fail. The misconception is that the t -test requires normally distributed scores, when in fact it only requires their mean to be approximately normal. The misuse is the reliance on the Wilcoxon test as a quick remedy when normality is questioned, because its own assumption of symmetry is actually far stronger, yet almost always ignored. Through analysis and simulation, we show that this oversight makes the Wilcoxon test fail catastrophically for common IR data, whereas the t -test remains robust. In short, the supposed ‘‘safe’’ alternative actually turns out to be the riskier choice, and continuing its use only deepens the confusion and undermines the credibility of our findings. 2 2 2 Data and code are available at https://github.com/julian-urbano/sigir2026-wilcoxon . 2. Systematic Review of the Statistics Literature In order to understand the confusion around statistical testing, we review a selection of textbooks to investigate how the Statistics literature describes these methods. In particular, we selected a total of 25 textbooks, categorized as follows: • 11 books on Statistics in general ( 7 ; 13 ; 22 ; 35 ; 36 ; 40 ; 47 ; 53 ; 61 ; 75 ; 80 ) . • 6 books specifically on Non-parametric Statistics ( 17 ; 32 ; 33 ; 46 ; 50 ; 62 ; 79 ) . • 8 books on Statistics for particular fields such as Behavioral Sciences ( 52 ) , Biology ( 66 ) , Business and Economics ( 1 ) , Computer Science ( 3 ) , Education ( 16 ) , Engineering ( 44 ) , Environmental Science ( 39 ) , and Psychology ( 2 ) . We examined and annotated all these sources with respect to how they discuss recommendations, non-parametric methods, the t -test, and the Wilcoxon test. 2.1. Recommendations When researchers seek guidance on which statistical test to use, many sources offer quick roadmaps with questions like “how many groups do you have?” or “are your data normally distributed?” to guide them. Of the 25 books we reviewed, 10 provided such shortcuts in the form of tables or decision trees (see Table 1 ). The first noteworthy observation is the diversity of descriptions about the object of the test leading to the paired-sample Wilcoxon test, ranging from vague statements like “differences between groups,” to precise ones such as “location,