Good practices for evaluation of machine learning systems

Luciana Ferrer, Odette Scharenborg, Tom Bäckström · 2024

Read Good practices for evaluation of machine learning systems by Luciana Ferrer, Odette Scharenborg, Tom Bäckström (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03700.txt.

Introduction

Luciana Ferrer, Odette Scharenborg, and Tom Bäckström write a 2024 guide to testing machine learning systems well. Many choices shape a test result, from training data to the score picked. The authors argue the test setup itself is the most fateful choice of all. A poor setup can make results fail on new data or miss the point of the task.

The guide walks through three parts of setup design. First comes data choice, then score choice, then how to judge whether gaps between models are real. Examples come from speech work, the authors' own field. A list of common mistakes closes each part. The guide aims to help teams plan their tests before they run a single trial.

Catalog record

CREATORLuciana Ferrer, Odette Scharenborg, Tom Bäckström
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03700

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Good practices for evaluation of machine learning systems v1.0: This document is a work in progress. New versions will be uploaded in the upcoming months. If you have constructive comments or suggestions, please write to [email protected]. Luciana Ferrer, Odette Scharenborg, Tom Bäckström Abstract Many development decisions affect the results obtained from ML experiments: training data, features, model architecture, hyperparameters, test data, etc. Among these aspects, arguably the most important design decisions are those that involve the evaluation procedure. This procedure is what determines whether the conclusions drawn from the experiments will or will not generalize to unseen data and whether they will be relevant to the application of interest. If the data is incorrectly selected, the wrong metric is chosen for evaluation or the significance of the comparisons between models is overestimated, conclusions may be misleading or result in suboptimal development decisions. To avoid such problems, the evaluation protocol should be very carefully designed before experimentation starts. In this work we discuss the main aspects involved in the design of the evaluation protocol: data selection, metric selection, and statistical significance. This document is not meant to be an exhaustive tutorial on each of these aspects. Instead, the goal is to explain the main guidelines that should be followed in each case. We include examples taken from the speech processing field, and provide a list of common mistakes related to each aspect. 1 Preliminaries Tasks and applications: In this document we define an “application” as a specific use-case scenario for a machine learning system, and a “task” as a group of applications that are closely related to each other. For example, speaker verification is a task for which there are many different applications. The application is what determines the type of data and the metric of interest. For example, in forensic applications of speaker verification systems, the cost of false positives (incorrectly predicting that the speakers in the two signals under comparison are the same) may be much higher than for speaker search applications where false negatives (missing to find a match) may be more costly than false positives. Further, even two speaker search scenarios may have different costs depending on how the end user is expected to interact with the system. In this paper, we consider those as two different applications. In summary: an application is a use-case scenario that completely determines the relevant data and performance metric. Systems and methods: We will consider two common experimental scenarios. In one case, we wish to evaluate a specific “system”. This system is already trained and it will be unchanged at time of deployment. We wish to know how that system will perform in practice, as-is. This is the case, for example, when presenting foundational models. We do not expect to retrain those models on different data. If we did, we would probably consider it a new system and evaluate it separately. Another common experimental scenario is one where the goal is to compare approaches or “methods”. In this case, the system is not frozen. We assume we might eventually train it on a different dataset. This is the case, for example, when we wish to make scientific conclusions about which features are more informative within some set of features of interest or when we wish to compare training procedures or architectures. In that scenario, we do not want our conclusions to be tied to a specific training dataset, so we wish to evaluate the method, rather than one specific system. As we will see, the difference between evaluating systems or evaluating methods will affect both the management of the data and the approach needed to compute confidence intervals 1 1 1 Dietterich [ 1 ] uses the terms classifiers and algorithms to refer to these concepts, here we use systems and methods for more generality. . 2 Datasets Let us first consider the simplest scenario where three sets are used for experimentation (see Figure 1 ): • a training set used for estimating the model’s parameters; • a development or validation set used for making system design decisions (hyperparameters, architectures, features); • and an evaluation or test set used for reporting the final performance, which should be a good predictor of the performance we will see during deployment. Training data Used for model training Development data Used for hyperparameter tuning Evaluation data Used for final performance assessment Figure 1: Data splits into three sets for training, development and evaluation. From these three sets, the evaluation set is the one that requires the most careful design. We want the performance metric computed on that set to be a good predictor of how the system will work in practice. To this end, the evaluation data should resemble as closely as possible the data that will be encountered during deployment. As long as that set is correctly selected, we are safe in the sense that if bad development decisions are made, we will see them reflected as a bad performance in the evaluation set. A discussion of these topics can be found in [ 12 ] . 2.1 The evaluation data In order for the evaluation results to be a good prediction of deployment results, the evaluation data should be as similar to the use-case data as possible. This means that the conditions of the samples should reflect those that the system will encounter when deployed. For speech processing tasks the conditions of a sample include the microphones used for recording, background noises, sample duration distribution, languages, etc. Importantly, if the system will be used on speakers that are not the same as the training speakers, the evaluation data should also correspond to speakers different from the training speakers. Further, if the system will be used in unknown acoustic conditions potentially different from those found in the training data, then it should also be tested on evaluation data with acoustic conditions that are not necessarily well represented in the training data - ideally, in a variety of such conditions to observe the spread of possible performances. In addition, the evaluation data should never be used for development. Selecting hyperparameters, features, and architectures using the same data on which the final performance is reported would result in an optimistic assessment of such performance. The evaluation data should ideally be held out during the whole development process and only uncovered at the very end, when all development decisions have been made, just as it is done in challenges. In this way, there is no risk of making any decisions based on these results. Finally, it is generally a good idea to have more than one evaluation set. Or, in other words, to divide the evaluation set into subsets of interest to better analyze how the system will perform in each of those subsets. For example, whenever possible, it is useful to report results on female vs. male, young vs. old, native vs. nonnative speakers, and any other slicing of the data that may be meaningful for the task. 2.2 The development data During development, the data can, in principle, be determined in any way you like since performance on that set is not expected to be predictive of future performance anyway. Yet, if the selection of

Read the complete text