Unsupervised optimal deep transfer learning for classification under general conditional shift

Junjun Lang, Yukun Liu · 2025

Read Unsupervised optimal deep transfer learning for classification under general conditional shift by Junjun Lang, Yukun Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12729.txt.

Introduction

Junjun Lang and Yukun Liu wrote this 2025 statistics paper on transfer learning when data shifts between a source and a target. A classifier trained only on labeled source data can mislead when used on unlabeled target data from a different mix. Transfer learning tries to carry knowledge across, but it often rests on strict terms such as label shift.

The authors set out a broader term, called general conditional shift, which holds label shift as one special case. Under that term, they show that both the target mix and the shift function can be pinned down from data. They estimate source label chances with deep neural networks. Then they move that estimate to the target and find the target label mix by a pseudo-likelihood route. The final classifier is built without ever having to estimate the shift function itself.

Catalog record

CREATORJunjun Lang, Yukun Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-12729

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unsupervised optimal deep transfer learning for classification under general conditional shift Junjun Lang, Yukun Liu Abstract Classifiers trained solely on labeled source data may yield misleading results when applied to unlabeled target data drawn from a different distribution. Transfer learning can rectify this by transferring knowledge from source to target data, but its effectiveness frequently relies on stringent assumptions, such as label shift. In this paper, we introduce a novel General Conditional Shift (GCS) assumption, which encompasses label shift as a special scenario. Under GCS, we demonstrate that both the target distribution and the shift function are identifiable. To estimate the conditional probabilities 𝜼 P {\mbox{\boldmath{$\eta$}}}_{P} for source data, we propose leveraging deep neural networks (DNNs). Subsequent to transferring the DNN estimator, we estimate the target label distribution 𝝅 Q {\mbox{\boldmath{$\pi$}}}_{Q} utilizing a pseudo-maximum likelihood approach. Ultimately, by incorporating these estimates and circumventing the need to estimate the shift function, we construct our proposed Bayes classifier. We establish concentration bounds for our estimators of both 𝜼 P {\mbox{\boldmath{$\eta$}}}_{P} and 𝝅 Q {\mbox{\boldmath{$\pi$}}}_{Q} in terms of the intrinsic dimension of 𝜼 P {\mbox{\boldmath{$\eta$}}}_{P} . Notably, our DNN-based classifier achieves the optimal minimax rate, up to a logarithmic factor. A key advantage of our method is its capacity to effectively combat the curse of dimensionality when 𝜼 P {\mbox{\boldmath{$\eta$}}}_{P} exhibits a low-dimensional structure. Numerical simulations, along with an analysis of an Alzheimer’s disease dataset, underscore its exceptional performance. Keywords: Transfer Learning; Deep Neural Network; General Conditional Shift; Classification; Minimax Optimality 1 Introduction Classifiers trained using labelled source data are typically employed to classify unlabelled target data that follow the same distribution. When the target data distribution deviates from that of the source data, the classifiers may lead to biased or misleading conclusions. Transfer learning is capable of correcting such classifiers by integrating information from both the labelled source data and the unlabeled target data. Generally speaking, transfer learning, also known as domain adaptation, arises in situations where the availability of data from the target domain is limited, whereas a considerable amount of data exists in a related source domain, thereby enabling knowledge transfer from the latter to the former. It has attracted significant attention across a wide variety of applications, particularly in computer vision ( Tzeng et al., 2017 ) , speech recognition ( Huang et al., 2013 ) , and machine learning ( Iman et al., 2023 ) . See Weiss et al. (2016) and Zhuang et al. (2020) for more information regarding transfer learning and its applications. This paper focuses on transfer learning within the framework of nonparametric classification and is committed to utilizing modern deep learning techniques to develop an efficient algorithm that adapts classifiers trained on the source to the target domain under a general distribution shift assumption. 1.1 Problem statement Let X\in\mathcal{X}\subset\mathbb{R}^{d} for d\geq 1 denote a vector of covariates or features and let Y\in\mathcal{Y}\subset\mathbb{R} represent a discrete outcome or label. Denote the joint distributions of (X,Y) from the source and target domains by P_{(X,Y)} and Q_{(X,Y)} , respectively. Suppose that we have labelled source data and unlabelled or unsupervised target data, \displaystyle(X_{1}^{P},Y_{1}^{P}),\cdots,(X_{n_{P}}^{P},Y_{n_{P}}^{P})\stackrel{{\scriptstyle i.i.d}}{{\sim}}P_{(X,Y)},\qquad X_{1}^{Q},\cdots,X_{n_{Q}}^{Q}\stackrel{{\scriptstyle i.i.d}}{{\sim}}Q_{X}, (1) where Q_{X} is the distribution function or probability measure of X in the target domain. Because the labels of the target data are unavailable, it is challenging, if not impossible, to learn a classifier with theoretical guarantee for the target data based solely on the target data themselves. This goal can be achieved by transferring information from the source data to the target data. A typical example is medical diagnosis, where X usually represents the physical indicators of patients that are relatively easy to measure, while Y measures the quality of the patient’s condition after treatment, which is often costly or difficult to measure. To save costs, doctors may wish to infer whether patients can end treatment based on the data of previous patients combined with the easily measurable physical indicators of the current patients. Transfer learning or domain adaptation is impossible without underlying assumptions ( Ben-David et al., 2010 ) . Its validity hinges on the similarity between the distributions of source and target data. Within this framework, two prominent similarity assumptions are covariate shift ( Kpotufe and Martinet, 2021 ; Sugiyama et al., 2008 ) and label shift ( Saerens et al., 2002 ; Maity et al., 2022 ) . In the scenario of covariate shift, the conditional distributions of Y given X remain consistent between the source and target data. Consequently, classifiers trained using source data yield valid predictions when directly employed on target data. Let P_{X\mid Y}(x\mid y) and Q_{X\mid Y}(x\mid y) be the probability density or mass functions of X given Y=y in the source and target domains, respectively. Conversely, label shift posits that P_{X\mid Y}(x\mid y)=Q_{X\mid Y}(x\mid y) and the marginal distributions of Y may diverge between the source and target domains. Label shift typically arises when external factors alter the distribution of labels. For instance, in medical diagnosis, symptoms (features) are often indicative of diseases (labels), but the distribution of diseases itself may shift. Despite its widespread popularity in the realm of transfer learning, the label shift assumption may be restrictive, prohibiting any alterations in the conditional distribution of X given Y . For instance, the advent of COVID-19 has demonstrated that, even when a patient’s disease is known, the distribution of their symptoms can change post-pandemic. To accommodate such variations, we propose relaxing the label shift assumption and adopting a more general distribution shift assumption: Assumption 1 (General Conditional Shift, GCS) . There exists a function class \mathscr{H} on \mathcal{X} containing the constant function 1 such that for a function h(\cdot)\in\mathscr{H} , \displaystyle\frac{Q_{X\mid Y}(x\mid y)}{P_{X\mid Y}(x\mid y)}=h(x),\quad\forall x\in\mathcal{X},\;y\in\mathcal{Y}. (2) The GCS assumption stipulates that the ratios of the conditional densities of X given Y=y in the target and source domains are equivalent to a common function h(x) for all values of y . When the function class \mathscr{H} comprises solely the element 1, this assumption simplifies to label shift. Furthermore, it can be perceived as a modification of the conditional shift proposed by Liu et al. (2021) or the exponential tilting model presented by Maity et al. (2023) . 1.2 Our contributions In this paper, we develop a minimax optimal classifier for unsupervised target data under the GCS assumption by leveraging knowledge transferred from the source domain using deep neural networks (DNNs). Our contributions are outlined as follows: 1. We introduce a novel distribution shift assumption, namely the GCS model presented in ( 2 ), which encompasses the traditional label shift assumption as a special case. The GCS model not only provides greater flexibility compared to label shift but also enables the identification and consistent estimation for the classification function of target domain by utilizing information from both source and target data. 2. Under the GCS assumption, we incorporate DNNs into transfer learning for classification tasks based on labeled source data and unlabeled target data. In contrast, traditional non-parametric methods, such as kernel-based techniques ( Maity et al., 2022 ) or the K-NN method ( Cai and Wei, 2021 ) , have been utilized for this purpose in the existing literature. Given the DNN estimator of {\boldsymbol{\eta}_{P}}(x) , the classification function of

Read the complete text