Robust Semi-Supervised Learning in Open Environments

Lan-Zhe Guo, Lin-Han Jia, Jie-Jing Shao, * · 2024

Read Robust Semi-Supervised Learning in Open Environments by Lan-Zhe Guo, Lin-Han Jia, Jie-Jing Shao, * (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18256.txt.

Introduction

Lan-Zhe Guo and co-authors survey ways to train AI models when labeled data is scarce but unlabeled data is plentiful, and the two sets do not match. Standard semi-supervised learning assumes both sets share the same labels, features, and patterns. In real open settings they often differ, and learning from mismatched data can make a model worse than using labels alone.

The paper sums up recent fixes sorted by the kind of mismatch: wrong or shifted labels, changed features, and different data patterns. It also presents the test sets used to judge these methods fairly. Open problems are listed at the end. The survey is a guide for researchers who need learning methods that stay safe when the unlabeled data cannot be fully checked by hand.

Catalog record

CREATORLan-Zhe Guo, Lin-Han Jia, Jie-Jing Shao, *
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robust Semi-Supervised Learning in Open Environments Lan-Zhe Guo, Lin-Han Jia, Jie-Jing Shao, * Abstract Semi-supervised learning (SSL) aims to improve performance by exploiting unlabeled data when labels are scarce. Conventional SSL studies typically assume close environments where important factors (e.g., label, feature, distribution) between labeled and unlabeled data are consistent. However, more practical tasks involve open environments where important factors between labeled and unlabeled data are inconsistent. It has been reported that exploiting inconsistent unlabeled data causes severe performance degradation, even worse than the simple supervised learning baseline. Manually verifying the quality of unlabeled data is not desirable, therefore, it is important to study robust SSL with inconsistent unlabeled data in open environments. This paper briefly introduces some advances in this line of research, focusing on techniques concerning label, feature, and data distribution inconsistency in SSL, and presents the evaluation benchmarks. Open research problems are also discussed for reference purposes. keywords machine learning, open environment, semi-supervised learning, robust SSL † † fcssetup: received = month dd, yyyy, accepted = month dd, yyyy, corr-email = [email protected], [1,3]Yu-Feng Li 1 Introduction Semi-supervised learning (SSL) is an effective learning paradigm to improve learning performance by attempting to exploit abundant unlabeled data when labels are scarce. It has been reported that, in certain cases, such as image classification, SSL methods can achieve the performance of purely supervised learning even when a substantial portion of the labels in a given data set has been discarded [ 1 ] . It is noticeable that the current success of SSL is mostly based on the close environment assumption where important factors between the labeled and unlabeled data are consistent. For example, all the unlabeled instances should belong to the class label set in labeled data, the features describing labeled and unlabeled data should be the same, and all labeled and unlabeled data should be sampled from an identical distribution. Figure 1 illustrates those consistent factors assumed in close environment SSL studies. Figure 1: Typical consistent factors assumed in close environment semi-supervised learning. However, many real-world applications involve open environments [ 2 ] where class label set, feature space, and data distribution could be inconsistent between labeled and unlabeled data. The main reason lies in the fact that the collection process of unlabeled data is different from labeled data, which lacks human supervision and easily collects data that are inconsistent with the target task. It is also impossible to manually validate the quality of unlabeled data, otherwise it goes against the SSL’s purpose of reducing human labor. It has been widely reported that SSL suffers severe performance degradation problems with inconsistent unlabeled data and could be even worse than the simple supervised learning baseline which does not exploit more unlabeled data [ 3 , 4 , 5 , 6 , 7 , 8 ] . Such phenomena undoubtedly violate the expectations of SSL and limit its effectiveness in more practical tasks. It seems the robust SSL in open environments is relevant to studies like out-of-distribution (OOD) detection [ 9 , 10 , 11 , 12 ] , or open set recognition [ 13 , 14 ] . However, these studies either assume that there is an accurate classification model or sufficient labeled data, which limits their application in SSL. There are also some unsupervised OOD detection studies [ 15 ] utilizing the power of the contrastive learning [ 16 ] framework to learn better representation for OOD detection. However, although these studies do not require labels, they still need a large amount of in-distribution data for training, while in open environment SSL it is difficult to get the clean in-distribution unlabeled data. Despite the grand challenges, many research efforts have recently been devoted to robust SSL in open environments. This paper will briefly introduce some advances in this line of research, focusing on approaches concerning inconsistent labels, inconsistent features, and inconsistent distributions between labeled and unlabeled data. Moreover, we introduce the benchmark dataset and performance measures applicable to evaluate the robustness of SSL in open environments and provide a public SSL toolkit for related research. Open research problems are also discussed for reference purposes. 2 Robust SSL in Open Environments In the SSL task, we are given a set of training data which includes labeled data set \mathcal{D}_{l} consists of n labeled instances \mathcal{D}_{l}=\{({\bf x}_{1},{\bf y}_{1}),\cdots,({\bf x}_{n},{\bf y}_{n})\} and unlabeled data set \mathcal{D}_{u} consists of m unlabeled instances \mathcal{D}_{u}=\{{\bf x}_{n+1},\cdots,{\bf x}_{n+m}\} . Usually, m\gg n , {\bf x}\in{\mathcal{X}}\in{\mathbb{R}}^{d} , {\bf y}\in{\mathcal{Y}}=\{1,\cdots,K\} where d is the number of feature dimension and K is the number of classes. The goal of SSL is to learn a model f({\bf x};\theta):\{{\mathcal{X}};\Theta\}\to{\mathcal{Y}} parameterized by \theta\in\Theta from training data to minimize the generalization risk R(f)={\mathbb{E}}_{(X,Y)}[\ell(f(X;\theta),Y)] , where \ell:{\mathcal{Y}}\times{\mathcal{Y}}\to{\mathbb{R}} refers to certain loss function, e.g., mean squared error or cross-entropy loss. In open environments, unlabeled data could be inconsistent with labeled data in terms of class label space, feature dimension, and data distribution. We denote the degree of inconsistency as t\in[0,1] . A higher t indicates a greater inconsistency, i.e., more unlabeled instances that are inconsistent with the target task. The robust SSL studies in open environments aim to decrease the negative impact of inconsistent unlabeled data, on the one hand, improve the performance via exploiting unlabeled data, and on the other hand, in the worst case, the SSL performance should not be worse than the supervised learning baseline which does not exploit more unlabeled data. 3 Label Inconsistent Close-environment SSL studies typically assume that the class label of any unlabeled instances should be a member of the given label space {\mathcal{Y}} . However, this assumption does not always hold. This is because unlabeled data is much easier to collect than labeled data in real-world applications, and the collection process of unlabeled data has less human verification. Thus, it is more likely for unlabeled data to have unseen classes that are irrelevant to the target task. For example, in the image classification task, unlabeled images crawled from Internet/social networking according to keywords usually contain broader category concepts than labeled data [ 17 , 5 , 6 , 18 ] . We illustrate label inconsistency in Figure 2 to help understand the problem. Figure 2: Irrelevant classes occur in the unlabeled dataset. Many researchers have pointed out that SSL is not robust to irrelevant unseen classes of unlabeled data, and could perform even worse than the simple supervised learning method that uses only a small number of labeled data [ 4 , 5 ] . The straightforward idea to deal with this problem is to detect and remove these irrelevant unseen class unlabeled instances. It is noteworthy that this problem is different from OOD detection since OOD detection approaches typically require a large corpus of in-distribution labeled data and would fail due to the scarcity of labeled data in SSL. Recently, a simple yet effective approach was proposed to tackle the semi-supervised OOD detection issue [ 19 ] . This approach learns a new representation space via a novel distance measure in which OOD samples could be separated well with limited labeled data and in-distribution data. Some approaches try to decrease the negative impact of these unseen class unlabeled instances in the training process. Various scoring

Read the complete text