Geodesic Flow Kernels for Semi-Supervised Learning on Mixed-Variable Tabular Dataset

Yoontae Hwang, Yongjae Lee · 2024

Read Geodesic Flow Kernels for Semi-Supervised Learning on Mixed-Variable Tabular Dataset by Yoontae Hwang, Yongjae Lee (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12864.txt.

Introduction

Yoontae Hwang and Yongjae Lee build GFTab, a way to learn from mixed tables of data when few rows are labeled. Their paper is from two thousand twenty-four.

Tables join smooth numeric fields with class fields, and their shape is hard for many tools to read. GFTab has three new parts. It mars each kind of field in a way that fits its type. It weighs how near two rows are by a flow measure on a curved space. And it uses tree-based codes to draw facts from the labeled rows. The team built a test set of twenty-one tables from varied fields. GFTab beat known tools on many of them, most of all when labels were scarce.

Catalog record

CREATORYoontae Hwang, Yongjae Lee
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-12864

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Geodesic Flow Kernels for Semi-Supervised Learning on Mixed-Variable Tabular Dataset Yoontae Hwang, Yongjae Lee Abstract Tabular data poses unique challenges due to its heterogeneous nature, combining both continuous and categorical variables. Existing approaches often struggle to effectively capture the underlying structure and relationships within such data. We propose GFTab (Geodesic Flow Kernels for Semi-Supervised Learning on Mixed-Variable Tabular Dataset), a semi-supervised framework specifically designed for tabular datasets. GFTab incorporates three key innovations: 1) Variable-specific corruption methods tailored to the distinct properties of continuous and categorical variables, 2) A Geodesic flow kernel based similarity measure to capture geometric changes between corrupted inputs, and 3) Tree-based embedding to leverage hierarchical relationships from available labeled data. To rigorously evaluate GFTab, we curate a comprehensive set of 21 tabular datasets spanning various domains, sizes, and variable compositions. Our experimental results show that GFTab outperforms existing ML/DL models across many of these datasets, particularly in settings with limited labeled data. Figure 1: The proposed model (GFTab) is a semi-supervised framework specifically designed for handling tabular data. Introduction One of the long-standing goals in ML has been to discover structure and pattern in data. While these efforts have been very successful in handling images, audio, and text, there is still room for improvement in handling tabular data. There are three main reasons why handling tabular data is difficult. One possible reason for this is that, unlike images and text, which share certain structural aspects within Euclidean space, tabular data do not appear to have such solid structural similarities. For instance, in an image, adjacent pixels are often observed to have similar color values and belong to the same object. However, in a tabular dataset, adjacent rows or columns may not necessarily have any inherent relationship or similarity. In most cases, the nature of tabular data would not change at all even if the order of rows and columns are shuffled. Second, tabular datasets may present certain challenges due to the presence of both continuous and categorical variables, each with distinct characteristics. Continuous variables are characterized by an infinite number of potential values within a specified range, whereas categorical variables are typically represented by values from a finite set. Moreover, while continuous variables can be ordered, categorical variables cannot be ordered in general. It would be beneficial to consider the inherent differences between these variable types ( Gorishniy et al. 2021 ) , rather than treating them identically as is often done in previous studies. Additionally, in real world, the proportion of continuous and categorical variables can vary considerably across different datasets, which can make it challenging to develop approaches that are widely applicable. This heterogeneity contributes to the domain-specific nature of tabular research and hinders the creation of universally applicable methods. It is possible that studies on tabular datasets have overlooked these complexities, evaluating performance only on benchmark data with predominantly continuous variables, which may not fully represent real-world scenarios. Finally, real-world tabular datasets often lack labels, which are crucial for supervised learning tasks. In the absence of labels, discovering meaningful patterns and structures within the data becomes an even more daunting task. Without labels to guide the learning process, the model must rely on the intrinsic structure and relationships within the data to uncover relevant patterns, which can be particularly difficult given the lack of inherent structural similarities and the heterogeneous nature of tabular datasets. Some studies ( Bahri et al. 2022 ; Ucar, Hajiramezanali, and Edwards 2021 ; Yoon et al. 2020 ) have demonstrated encouraging outcomes in addressing tabular datasets with limited labeled data. These methods employ corruption techniques to maximize the similarity between the original input and corrupted versions, thereby enabling the models to learn meaningful representations. However, these methods disregard variable types and have primarily been evaluated on datasets containing only continuous variables. Consequently, their effectiveness may be limited when dealing with real-world tabular datasets that often contain a mix of continuous and categorical variables ( Borisov et al. 2022 ) . In this paper, we introduce GFTab: G eodesic F low Kernel on a Mixed-variable Tab ular dataset, an approach designed to address the unique challenges inherent in tabular data, particularly in semi-supervised learning settings. GFTab consists of three key components: • Variable-specific corruption methods that account for the distinct characteristics of continuous and categorical variables. • Geodesic flow kernel that captures the geometric relationships between corrupted representations on the Grassmann manifold. • Tree-based embedding technique that leverages labeled data to learn hierarchical feature relationships. By integrating these components, GFTab is able to effectively learn robust representations of tabular data, even in scenarios with limited labeled samples or noisy labels. Our framework is designed to be applicable across a wide range of tabular datasets and domains. In the following sections, we describe each component of GFTab in detail, present our proposed benchmark datasets for evaluating tabular learning methods, and demonstrate GFTab’s effectiveness through extensive experiments. Related works Tabular DL Models. Due to the importance of tabular data in real-world industries, ML/DL models for tabular data have been actively studied recently. The TabTransformer ( Huang et al. 2020 ) generated embeddings on categorical variables to learn the context of tabular data in a robust way, and it performed well in terms of prediction Accuracy. However, they only used a simple MLP layer for continuous variables. AutoInt ( Song et al. 2019 ) is designed to specifically learn the interactions between features in high-dimensional tabular data sets. The FT-transformer ( Gorishniy et al. 2021 ) used Transformer layers, but it treated all features in the same way regardless of their variable types (categorical and continuous). Furthermore, it becomes computationally inefficient when there are a large number of features. NODE ( Popov, Morozov, and Babenko 2020 ) proposed a method for training deep ensembles of oblivious differentiable decision trees on tabular data and demonstrated strong performance on only a few datasets. The TabGNN ( Guo et al. 2021 ) modeled the relationships within the tabular data samples using graph neural networks in an explicit and systematic manner. However, its numerical experiments are limited to large-sized tabular datasets only. NPTs ( Kossen et al. 2021 ) used self-attention to explicitly infer the relationships between data points like TabGNN. However, it is computationally expensive and has only been demonstrated on a small number of datasets. The GRANDE ( Marton et al. 2024 ) model proposed an ensemble learning method to integrate multiple neural networks. TabPFN ( Hollmann et al. 2023 ) introduced a probabilistic approach for tabular data, leveraging Bayesian principles to provide robust uncertainty estimates and improve prediction accuracy. However, to the best of our knowledge, most of the models discussed above have been tested using datasets with 100% labels, which is not representative of real world applications. Tabular Self- and Semi-Supervised Learning Self-supervised learning techniques have been actively studied in computer vision ( Chen et al. 2020 ; Jing and Tian 2020 ; Zhai et al. 2019 ) and natural language processing ( Lee, Chang, and Toutanova 2019 ; Qiu et al. 2020 ; Ruder and Plank 2018 ; Song et al. 2020 ) , but not much research has been conducted for tabular data. The TABBIE ( Iida et al. 2021 )

Read the complete text