VisTabNet: Adapting Vision Transformers for Tabular Data
Witold Wydmański, Ulvi Movsum-zada *, Jacek Tabor *, Marek Śmieja * · 2025
Read VisTabNet: Adapting Vision Transformers for Tabular Data by Witold Wydmański, Ulvi Movsum-zada *, Jacek Tabor *, Marek Śmieja * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-00057.txt.
Introduction
Witold Wydmanski and colleagues adapt image-trained vision transformers to work on tables of numbers, the most common data form in biology, industry, and finance. Deep learning has transformed text and image work, yet small tabular data sets still favor older ensemble methods, and training a fresh network for each small table is wasteful. Writing with a twenty twenty-five arXiv release, the team projects each table row into patch-like tokens. A pretrained vision transformer can accept those tokens. The team then runs the borrowed image encoder unchanged on them.
On several small data sets with under a thousand rows, the borrowed model beat both classic ensembles and recent purpose-built deep models. No new architecture had to be designed for tables. The authors present the result as a widening of transfer learning: skills learned from pictures can carry to spreadsheets.
Catalog record
| CREATOR | Witold Wydmański, Ulvi Movsum-zada *, Jacek Tabor *, Marek Śmieja * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-00057
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VisTabNet: Adapting Vision Transformers for Tabular Data Witold Wydmański, Ulvi Movsum-zada *, Jacek Tabor *, Marek Śmieja * Abstract Although deep learning models have had great success in natural language processing and computer vision, we do not observe comparable improvements in the case of tabular data, which is still the most common data type used in biological, industrial and financial applications. In particular, it is challenging to transfer large-scale pre-trained models to downstream tasks defined on small tabular datasets. To address this, we propose VisTabNet – a cross-modal transfer learning method, which allows for adapting Vision Transformer (ViT) with pre-trained weights to process tabular data. By projecting tabular inputs to patch embeddings acceptable by ViT, we can directly apply a pre-trained Transformer Encoder to tabular inputs. This approach eliminates the conceptual cost of designing a suitable architecture for processing tabular data, while reducing the computational cost of training the model from scratch. Experimental results on multiple small tabular datasets (less than 1k samples) demonstrate VisTabNet’s superiority, outperforming both traditional ensemble methods and recent deep learning models. The proposed method goes beyond conventional transfer learning practice and shows that pre-trained image models can be transferred to solve tabular problems, extending the boundaries of transfer learning. We share our example implementation as a GitHub repository available at https://github.com/wwydmanski/VisTabNet . 1 Introduction Deep learning has achieved tremendous success in various domains, including natural language processing (NLP) [ 34 ] , computer vision (CV) [ 16 ] , and reinforcement learning (RL) [ 20 ] . Transformers, in particular, have become one of the most prominent neural architectures in NLP [ 31 ] and CV [ 7 ] , demonstrating remarkable improvements through their self-attention mechanism that captures global dependencies across inputs. In consequence, it is not surprising that several works focus on applying transformers beyond CV and NLP domains. In real-world applications, tabular data remains one of the most common data types, used extensively in biology [ 29 ] , medicine [ 25 ] , finance [ 5 ] , and manufacturing [ 22 ] . Recent reports indicate that data science practitioners work with tabular data as frequently as with texts or images 1 1 1 https://www.statista.com/statistics/1241924/worldwide-software-developer-data-uses/ . This is reflected in Kaggle’s dataset distribution 2 2 2 Statistics gatherer in 2023 from https://www.kaggle.com/datasets , where 6,688 datasets are tagged as ”tabular”, compared to 4,908 tagged as ”image” and 178 as ”text”. Despite the prevalence of tabular data in practical applications, deep learning models have yet to demonstrate significant improvements over traditional ensemble methods like XGBoost and Random Forests in this domain [ 3 , 15 ] . The heterogeneous nature of tabular data and typically small sample sizes pose particular challenges for deep learning approaches [ 11 , 19 ] . While various deep learning approaches, including transformer architectures [ 9 ] and hypernetworks [ 32 ] , have been adapted for tabular data, pre-training and transferring these models to downstream tasks remains challenging [ 35 ] . In this paper, we propose a novel approach to cross-modal transfer learning, reusing the Vision Transformer (ViT) for tabular data tasks; see Figure 1 . In contrast to the typical reasoning behind transferring a feature extractor inside the same domain, we explore cross-modal transfer. More precisely, we take the Transformer Encoder of ViT pre-trained on image data and introduce an adaptation network that maps tabular inputs to a form compatible with the pre-trained ViT Encoder, enabling the construction of meaningful representations while avoiding the computational cost of training from scratch. Our VisTabNet model demonstrates superior performance across multiple conventional tabular datasets ( Table 1 ) and shows effective few-shot transfer capabilities ( Figure 2 ). Through extensive experimental analysis ( Section 4.2 ), we provide insights into the effectiveness of cross-modal transfer and practical applications of VisTabNet. Our contributions are summarized as follows: • We propose a novel idea for cross-modal transfer learning, enabling the use of intrinsic patterns from one data modality to enhance training efficiency in another. • We introduce VisTabNet, a novel deep learning algorithm that leverages the middle layers of a Vision Transformer architecture to process tabular data. • We perform a comprehensive benchmark of VisTabNet against widely used shallow and deep learning methods across multiple diverse datasets, demonstrating its performance in learning from small datasets. 2 Related Work Tabular data is one of the most prevalent mediums in the world, right next to natural language, with over 5400 datasets present in OpenML [ 30 ] alone. For comparison, the most common NLP task in the Huggingface Dataset [ 17 ] repository, text classification, is present as a tag in just 2300 datasets. In contrast to computer vision or natural language processing, shallow models, such as Support Vector Machines [ 6 ] , Random Forests [ 2 ] and Gradient Boosting [ 8 ] , are usually the first choice for learning from tabular datasets. In particular, the family of Gradient Boosting algorithms [ 8 ] , including XGBoost [ 3 ] , LightGBM [ 14 ] , and CatBoost [ 24 ] , achieve impressive performance and frequently exceed the performance of deep learning models. Both Gradient Boosting as well as Random Forests generate an ensemble of weak learners composed of decision trees, but they differ in the way those trees are built and combined. To take advantage of the flexibility of neural networks, various architectures have recently been proposed to improve their performance on tabular data. Inspired by CatBoost, NODE performs a gradient boosting of oblivious decision trees, which is trained end-to-end using gradient-based optimization [ 23 ] . The aim of Net-DNF is to introduce an inductive bias in neural networks corresponding to logical Boolean formulas in disjunctive normal forms [ 13 ] . It encourages localized decisions, which involve small subsets of features. TabNet uses a sequential attention mechanism to select a subset of features, which are used at each decision step [ 1 ] . Hopular is a deep learning architecture in which every layer is composed of continuous modern Hopfield networks [ 27 ] . The Hopfield modules allow one to detect various types of dependencies (feature, sample, and target) and have been claimed to outperform concurrent methods on small and medium-sized datasets. The authors of [ 12 ] show that the key to boosting the performance of deep learning models is the application of various regularization techniques. They demonstrate that fully connected networks can outperform competitive techniques by applying an extensive search of possible regularizers. The authors of [ 9 ] introduced modified versions of ResNet and Transformer and showed that the latter outperforms previous neural network models on large datasets. In follow-up papers, the authors worked to transfer the constructed transformer model to other tabular datasets [ 35 ] . Although multiple authors of recent deep learning models often claim to outperform shallow ensemble models, other experimental studies seem to deny these conclusions, showing that typical ensemble methods with careful hyperparameter tuning still presents superior performance [ 10 , 28 ] . The authors of [ 11 , 19 ] investigated the situations when deep networks outperform gradient-boosted trees. Figure 1: Data flow architecture in VisTabNet. The tabular input is transformed