Transductive Model Selection under Prior Probability Shift

Lorenzo Volpi, Alejandro Moreo, Fabrizio Sebastiani · 2025

Read Transductive Model Selection under Prior Probability Shift by Lorenzo Volpi, Alejandro Moreo, Fabrizio Sebastiani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22647.txt.

Introduction

Lorenzo Volpi, Alejandro Moreo, and Fabrizio Sebastiani wrote this machine learning paper, published in 2025. It deals with transductive learning, where the unlabeled data a model must label are known ahead of time, during training.

The paper tackles model selection, the step of tuning a model's settings, when the class balance in the data has shifted. Standard tuning relies on cross-checks within the labeled training data, which can mislead under such a shift. The authors propose tuning the settings directly on the unlabeled target data instead. Tests show this targeted approach beats the standard route. The work matters for tasks like sorting documents or screening cases, where the mix of classes in new data rarely matches the training mix.

Catalog record

CREATORLorenzo Volpi, Alejandro Moreo, Fabrizio Sebastiani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-22647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Transductive Model Selection under Prior Probability Shift Lorenzo Volpi, Alejandro Moreo, Fabrizio Sebastiani Abstract Transductive learning is a supervised machine learning task in which, unlike in traditional inductive learning, the unlabelled data that require labelling are a finite set and are available at training time. Similarly to inductive learning contexts, transductive learning contexts may be affected by dataset shift, i.e., may be such that the IID assumption does not hold. We here propose a method, tailored to transductive classification contexts, for performing model selection (i.e., hyperparameter optimisation) when the data exhibit prior probability shift, an important type of dataset shift typical of anti-causal learning problems. In our proposed method the hyperparameters can be optimised directly on the unlabelled data to which the trained classifier must be applied; this is unlike traditional model selection methods, that are based on performing cross-validation on the labelled training data. We provide experimental results that show the benefits brought about by our method. K eywords Model selection \cdot Hyperparameter optimisation \cdot Classifier accuracy prediction \cdot Dataset shift \cdot Prior probability shift \cdot Transductive learning 1 Introduction Consider the outbreak of an epidemic, in which the prevalence of individuals affected by an infectious disease rapidly increases while the distribution of the symptoms (i.e., the effects) across the affected individuals remains unchanged. In such a scenario, we may want to train a classifier that, from the symptoms an individual displays, infers whether she is affected or not from the disease. We consider the situation in which (i) the data to be classified arrives in successive batches u_{1} , u_{2} , …, and, (ii) given the epidemic, we may expect the prevalence of the affected individuals to evolve rapidly, i.e., exhibit different values in different batches. Assume that, for training the classifier, and in particular for selecting the combination of hyperparameters expected to yield the best accuracy under the new (i.e., epidemic) conditions, we have access to training data collected in the old (i.e., pre-epidemic) conditions. This scenario is problematic, as the training data and the unlabelled data to be classified are not identically and independently distributed (IID), due to the fact that the prevalence of the individuals affected by the disease has changed from the training data to the unlabelled data. The classifier, and the chosen hyperparameter combination, may thus reveal suboptimal once used on the unlabelled data. In this paper we present a method for optimising the hyperparameters (i.e., for performing model selection – MS) directly on the batch of unlabelled data that need to be classified . For reasons that will be explained in Section 2 , we call this task transductive model selection (TMS). A TMS method has obvious advantages over the standard inductive model selection (IMS) method (that relies on cross-validation on the training data), since the chosen hyperparameters are tailored to the batch U_{i} of unlabelled data that need to be classified, and can thus deliver better performance on U_{i} than hyperparameters chosen via standard IMS. Our proposed method is based on techniques for classifier accuracy prediction (CAP) under dataset shift [ 6 , 9 , 18 ] , and essentially consists of (i) predicting the accuracy that different classifiers, instantiated with different choices of hyperparameters, would obtain on our “out-of-distribution” batch U_{i} , and (ii) classifying the data in batch U_{i} using the classifier whose predicted accuracy is highest. In other words, our TMS method replaces traditional accuracy computation on labelled data with accuracy estimation on unlabelled data. While the method is generic, we here restrict our attention to the case in which the data are affected by prior probability shift (PPS), an important type of violation of the IID assumption. For high-stakes applications such as the healthcare-related classification task discussed above, this suggests a policy of (a) training, on the labelled data, multiple classifiers, each characterised by a different combination of hyperparameters, 1 1 1 Note that this step is performed anyway during traditional hyperparameter optimisation. (b) storing these classifiers for later use, and, every time a new batch U_{i} of unlabelled data becomes available, (c) estimating (via CAP techniques) the accuracy that the different classifiers would have on U_{i} and (d) classifying the data in U_{i} via the classifier whose estimated accuracy is highest. In this way, assuming that Step (b) can be carried out efficiently, the classification of newly arrived data can be performed immediately and, as we will show, with a much higher accuracy than can be obtained via the traditional model selection method. Note that Step (a) is carried out only once, since we do not assume new labelled data to become available during the process. The rest of the paper is organised as follows. Section 2 introduces the notation and provides a detailed description of the proposed method. Section 3 presents the experiments we have carried out and discusses the results we have obtained. Section 4 concludes the paper with a summary of our findings and a discussion of potential applications of this method. 2 Transductive Model Selection under Prior Probability Shift In supervised machine learning we use training data to learn the internal parameters of the model (e.g., the weights of a neural network, or the coefficients of a hyperplane in an SVM). Many models trained in this way also rely on a set of hyperparameters (e.g., the learning rate in neural training, or the trade-off between margin and training error in SVMs) that impose higher-level constraints on the learning process. Unlike the internal parameters of the model, hyperparameters are not learned during training, but must be set in advance. Finding good values for the hyperparameters is crucial for achieving good performance. Model selection, the task of choosing the values of the hyperparameters, is typically carried out by (a) testing the accuracy of the model under different combinations of hyperparameter values using cross-validation on labelled data, and (b) choosing the combination of hyperparameter values that maximizes model accuracy. Relying on labelled data to evaluate differently configured models requires the labelled data to be representative of the unlabelled data the trained model will be applied to, a distributional assumption typically referred to as the IID assumption. Unfortunately, in real-world problems this assumption is often violated; in this case, the training data are not representative of the unlabelled data (which are thus said to be “out-of-distribution” data), and we say that the problem is affected by dataset shift [ 16 ] . In problems characterised by dataset shift, cross-validation on training data is thus a biased estimator of model accuracy [ 17 ] , and this often leads to suboptimal choices of the hyperparameter values. In classification, one type of dataset shift of particular relevance is prior probability shift (PPS) [ 16 ] , also known as label shift [ 11 ] . This type of shift (sometimes considered the “paradigmatic” case of dataset shift in classification [ 20 ] ) is characteristic of anti-causal learning problems [ 14 ] (also known as Y\rightarrow X problems [ 2 ] , where Y is a random variable ranging on the class labels and X is a random variable ranging on vectors of covariates), i.e., problems where the goal is to predict the causes of a phenomenon from

Read the complete text