A Unified Data Representation Learning for Non-parametric Two-sample Testing

Xunye Tian, Liuhua Peng, Zhijian Zhou, Mingming Gong, Arthur Gretton, Feng Liu · 2024

Read A Unified Data Representation Learning for Non-parametric Two-sample Testing by Xunye Tian, Liuhua Peng, Zhijian Zhou, Mingming Gong, Arthur Gretton, Feng Liu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00613.txt.

Introduction

Xunye Tian and co-authors present a 2024 paper on a basic test in data work. The test asks whether two sets of data come from the same source. Good tests depend on finding a clear way to show the data. Past work split data into a training set and a test set, and learned that view from training data alone.

The authors show that the full set can shape the view, as long as set labels stay hidden. Their method first learns the shape of the data, then trains a model to stress the gaps. In tests, the method beat well-known rivals. It used both the data shape and the group gaps.

Catalog record

CREATORXunye Tian, Liuhua Peng, Zhijian Zhou, Mingming Gong, Arthur Gretton, Feng Liu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00613

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Unified Data Representation Learning for Non-parametric Two-sample Testing Xunye Tian, Liuhua Peng, Zhijian Zhou, Mingming Gong, Arthur Gretton, Feng Liu Abstract Learning effective data representations has been crucial in non-parametric two-sample testing. Common approaches will first split data into training and test sets and then learn data representations purely on the training set . However, recent theoretical studies have shown that, as long as the sample indexes are not used during the learning process, the whole data can be used to learn data representations, meanwhile ensuring control of Type-I errors. The above fact motivates us to use the test set (but without sample indexes) to facilitate the data representation learning in the testing. To this end, we propose a representation-learning two-sample testing (RL-TST) framework. RL-TST first performs purely self-supervised representation learning on the entire dataset to capture inherent representations (IRs) that reflect the underlying data manifold. A discriminative model is then trained on these IRs to learn discriminative representations (DRs), enabling the framework to leverage both the rich structural information from IRs and the discriminative power of DRs. Extensive experiments demonstrate that RL-TST outperforms representative approaches by simultaneously using data manifold information in the test set and enhancing test power via finding the DRs with the training set. 1]Faculty of Engineering and Information Technology, University of Melbourne, AU 2]School of Mathematics and Statistics, University of Melbourne, AU 3]Gatsby Computational Neuroscience Unit, University College London, UK 1 Introduction Two-sample tests aim to answer a question: “Are two samples drawn from the same distribution?”. Classical two-sample tests, including t -tests which test the empirical mean differences between two samples, often need to assume that samples are drawn from specific distributions (e.g., Gaussian distributions with the same variance). To alleviate the assumptions, non-parametric two-sample tests are proposed to solve the problem only based on observed data [ Gretton et al., 2012b , Heller and Heller, 2016 , Székely and Rizzo, 2013 , Jitkrittum et al., 2016 , Chen and Friedman, 2017 , Ghoshdastidar et al., 2017 , Lopez-Paz and Oquab, 2017 , Ramdas et al., 2017 , Sutherland et al., 2017 , Gao et al., 2018 , Ghoshdastidar and von Luxburg, 2018 , Lerasle et al., 2019 , Liu et al., 2020 , Kirchler et al., 2020 , Kübler et al., 2020 , Cheng and Xie, 2021 , Kübler et al., 2022 , Kübler et al., 2022 , Liu et al., 2021 , Deka and Sutherland, 2023 , Bonnier et al., 2023 , Schrab et al., 2023 , Biggs et al., 2023 ] . For example, the Kolmogorov-Smirnov (K-S) test is designed to compare the cumulative distribution functions derived from two samples, but generalisation to higher dimension is challenging [ Bickel, 1969 ] . The maximum mean discrepancy (MMD) test adopts the kernel mean embedding of distribution and uses it to measure the discrepancy between two distributions Gretton et al. [2012a] whose dimensions can be relatively higher than classical methods [ Liu et al., 2020 ] . The statistics used in these non-parametric two-sample tests are also widely adopted in many other fields, such as domain adaptation, causal discovery, generative modeling, adversarial learning, and more [ Gong et al., 2016 , Bińkowski et al., 2018 , Stojanov et al., 2019 , Cano and Krawczyk, 2020 , Oneto et al., 2020 , Gao et al., 2021 , Fang et al., 2021b , Zhong et al., 2021 , Fang et al., 2021a , Song et al., 2021a , Tahmasbi et al., 2021 , Taskesen et al., 2021 , Bergamin et al., 2022 ] . To improve the test power of non-parametric two-sample tests in practical applications, recent studies have shown that learning good data representations is crucial before performing two-sample testing [ Kirchler et al., 2020 , Liu et al., 2020 , Liu et al., 2021 , Gao et al., 2021 , Bergamin et al., 2022 ] . For example, Kirchler et al. [2020] directly use a pre-trained feature extractor to extract features of two samples and find it is useful to increase the test power during the testing. Meanwhile, Liu et al. [2020] propose a learning paradigm to learn deep-net representations of data via maximizing the test power of MMD and show that the learned representations can help capture the difference between two complex-structured samples. Even though utilizing a fraction of the samples to train a classifier [ Lopez-Paz and Oquab, 2017 ] or a kernel function [ Liu et al., 2020 ] enables deriving discriminative representations (DRs) of the remaining samples, the data splitting process results in a trade-off between the extra power provided by the learned functions/kernels and the sacrificed power due to the decreasing testing sample size. However, Biggs et al. [2023] have pointed out that, after discarding the sample information (namely, we do not know which sample the data belongs to), learning purely inherent representations (IRs) from aggregated samples will not influence the type I error of permutation-based testing methods, but the missing discriminative power makes it underperform on the complex-structured data, which can further justify the correctness of learning good representations for testing. Motivated by the above theoretical studies and existing challenges, we propose a representation-learning two-sample testing (RL-TST) framework that focuses on learning good representations on the samples, from both IRs and DRs. Since two-sample testing data mainly follows a manifold assumption where the (high-dimensional) data lie (roughly) on a low-dimensional manifold, we could firstly learn an encoder from the representation learning that is responsible for extracting the IRs of entire samples . Then, train a discriminative model on the learned IRs will enable the model with discriminative ability directly on the inherent manifolds of samples rather than on the complex embedded space of samples. This framework captures the sample structure information discarded in the data splitting process and exhibits a higher discriminative power than purely unsupervised representation learning on the entire dataset. We conduct extensive experiments to implement RL-TST on different kinds of MMD-based two-sample testing methods, we verify the empirical effectiveness of RL-TST over the state-of-the-art (SOTA) two-sample testing methods on synthetic high-dimensional Gaussian mixture (HDGM) dataset, MNIST dataset and ImageNet dataset. These are the commonly used benchmarks to detect the performance of two-sample testing methods. Our main contributions are: • We propose a novel RL-TST, which can address the challenges of two existing frameworks and provide a new research direction of two-sample testing. • Empirically, the various implementations of RL-TST show their outperformances over different SOTA baselines across different benchmarks. • Comparatively, we provide the discussion and empirical evidence of why alternative frameworks, such as semi-supervised learning or purely self-supervised learning are facing challenges in two-sample testing scenarios. 2 Preliminaries Two-sample Testing. Two-sample testing is one of the statistical hypothesis tests that aims to assess whether two independent and identically distributed samples, denoted by S_{\mathbb{P}}=\{x_{i}\}_{i=1}^{n}\sim\mathbb{P}^{n} and S_{\mathbb{Q}}=\{y_{j}\}_{j=1}^{m}\sim\mathbb{Q}^{m} , where x_{i},y_{j}\in\mathcal{X} , are drawn from the same distribution [ Lehmann and Romano, 2005 ] . In two-sample testing, the null hypothesis H_{0} refers to two samples being drawn from the same distribution, which corresponds to \mathbb{P}=\mathbb{Q} . The alternative hypothesis H_{1} indicates that two samples are drawn from different distributions, meaning \mathbb{P}\neq\mathbb{Q} . Whether we should accept or reject H_{0} depends on the test statistic

Read the complete text