Performance Consistency of Learning Methods for Information Retrieval Tasks
Meng Yuan, Justin Zobel · 2025
Read Performance Consistency of Learning Methods for Information Retrieval Tasks by Meng Yuan, Justin Zobel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20804.txt.
Introduction
A search system that scores well once may score poorly when the training seed changes. Meng Yuan and Justin Zobel measure that instability in a 2025 paper on information retrieval methods. They test both classic statistical models and transformer-based learning models across three retrieval tasks.
Bootstrapping test sets gives one view of performance variation, and the authors confirm it works. Changing random seeds gives another view for learning methods. Statistical models stay stable under seed changes. Transformer models swing widely. In most cases their F1 scores varied by large margins across seeds. The authors warn that single-run results from such models can mislead, and that performance claims should report this seed-driven spread.
Catalog record
| CREATOR | Meng Yuan, Justin Zobel |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20804
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Performance Consistency of Learning Methods for Information Retrieval Tasks Meng Yuan, Justin Zobel Abstract A range of approaches have been proposed for estimating the accuracy or robustness of the measured performance of IR methods. One is to use bootstrapping of test sets, which, as we confirm, provides an estimate of variation in performance. For IR methods that rely on a seed, such as those that involve machine learning, another approach is to use a random set of seeds to examine performance variation. Using three different IR tasks we have used such randomness to examine a range of traditional statistical learning models and transformer-based learning models. While the statistical models are stable, the transformer models show huge variation as seeds are changed. In 9 of 11 cases the F1-scores (in the range 0.0–1.0) had a standard deviation of over 0.075; while 7 of 11 precision values (also in the range 0.0–1.0) had a standard deviation of over 0.125. This is in a context where differences of less than 0.02 have been used as evidence of method improvement. Our findings highlight the vulnerability of transformer models to training instabilities and moreover raise questions about the reliability of previous results, thus underscoring the need for rigorous evaluation practices. 1 Introduction In information retrieval (IR), a standard approach to evaluation of a method is to apply the method to a set of test data and report a numerical score. A simple example of a task where this kind of approach is used is binary text classification. If the method is based on some form of machine learning (a terminology used here in a broad sense, encompassing transformers as well as statistical methods), training data is used to establish the classifier, which is then applied to test data, producing a set of outputs that are either true or false. These outputs are then translated to measures such as precision or the F1-score. A typical element of such evaluations is that the score is reported to three or four decimal places without consideration given to the possibility of uncertainty, as might be measured by confidence or deviation. However, uncertainty can arise in a range of ways. One is the selection of test data. Statistical methods such as bootstrapping provide a way to estimate score variability arising from composition of the test data, and have been occasionally (if infrequently) used in such evaluations. Here we report bootstrap results on three classification tasks and nine ML methods, confirming its value as an indicator of accuracy. Another source of uncertainty for ML methods is the choice of seed, which influences multiple aspects of model training including the ordering of training data, initialization of parameters, and update order. This uncertainty is the primary focus of our work. In statistical machine learning, such as linear regression, training is fully deterministic: every use of a random seed can be traced, and the entire process can be reconstructed to understand precisely how parameters were updated. However, this determinism does not hold in deep learning. Even small differences in factors such as ordering of training data can in principle yield substantial performance variations. That is, the training process itself may to some extent be unstable, but it is rare for papers to acknowledge this instability, let alone test or report it ( Zhuang et al. 2022 ) . Such instability undermines reproducibility and indeed the usability of the methods. It also means that, without statistical examination or acknowledgement of variability, research results on these methods are not reliable – particularly as researchers in some cases choose random seeds to optimise performance. These concerns are not new; they have been examined for specific methods in several prior works ( Antoniak and Mimno 2021 ; Dutta et al. 2022 ; Fellicious et al. 2020 ; Isensee et al. 2024 ; Schader et al. 2024 ) . However, to our knowledge these issues have not been previously examined across a basket of ML methods in the context of IR tasks. We examine how seed choices affect model performance for nine statistical ML and transformer-based approaches on three representative IR tasks: sentiment analysis, fake news detection, and binary question answering. The statistical methods yield consistent results but the transformer-based methods show such high levels of variation that in many cases the ‘true’ results are effectively unknown to even a single decimal place. These results demonstrate that it is critical that researchers adopt more robust research practices. 2 Related work A commonly used approach for evaluation of the robustness of algorithms and metrics is bootstrapping, a statistical method for assessing the variance in a data set that has the benefit of not relying on distributional assumptions. The principle of bootstrapping is that a dataset of n items is sampled n times with replacement, creating a new dataset that is consistent in class distribution with the original data. Test set bootstrapping has been used for IR metric evaluation. For example, Sakai 2006 has used bootstrapping to estimate the minimum performance gap required for reliable discrimination between systems. Smucker et al. 2007 then adopted Sakai 2006 ’s approach to test three statistical methods for evaluating measure sensitivity: bootstrapping, permutation, and the t-test. Later work further explored bootstrap-based confidence intervals for rank correlation among metrics ( Sakai 2007 ) and studied evaluation stability under different sampling and pooling strategies involving bootstrapping ( Urbano et al. 2013 ) . Collectively, these studies establish bootstrapping as a principled approach for assessing both the discriminative power of IR evaluation measures and the robustness of experimental conclusions in IR. The use of random seeds for evaluation of ML variation is more diverse, in part because random seeds are involved in a variety of stages in model training and evaluation. For example, they help ML methods to avoid falling into local optima and therefore help avoid bias on model performance ( Goodfellow et al. 2016 ) . Random seeds have been examined in many previous works on ML evaluation, and have been considered as a common approach for robustness testing of machine learning algorithms ( Reimers and Gurevych 2017 ; Henderson et al. 2018 ; Dodge et al. 2019 ) . Over time this became a convention in the evaluation process and many recent works do not report the variance of performance with respect to multiple random seeds; instead, only an average is given. Several recent papers have explored concerns about the randomness involved in model training, especially for deep learning models ( Fellicious et al. 2020 ; Dutta et al. 2022 ; Antoniak and Mimno 2021 ; Schader et al. 2024 ) . The training process of deep learning models is to some extent non-deterministic due to a variety of components including the internal layers and GPU optimisation ( Zhuang et al. 2022 ) ; these components contribute to the randomness of the training process along with the random seed itself. Although quantitative studies on effects of random seeds in deep learning models are limited, a couple of recent studies have explored them from a variety of perspectives. Fellicious et al. 2020 examined the performance of ANN models with three optimisers using different random seeds and observed significant variance in the performance of the models, ranging