MultiTab: A Scalable Foundation for Multitask Learning on Tabular Data

Dimitrios Sinodinos, Jack Yi Wei, Narges Armanfard · 2025

Read MultiTab: A Scalable Foundation for Multitask Learning on Tabular Data by Dimitrios Sinodinos, Jack Yi Wei, Narges Armanfard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09970.txt.

Introduction

Dimitrios Sinodinos, Jack Yi Wei, and Narges Armanfard present MultiTab-Net, a transformer built for multi-task learning on tabular data, the rows-and-columns format behind finance, health, and commerce systems. Multi-task learning can share strength across related targets. But most tabular work sits in product-ranking systems. It leans on perceptron backbones that capture feature interactions weakly and scale poorly.

Their design uses a multi-task masked-attention mechanism that models feature relationships dynamically while limiting competition between tasks. Across ranking, census-like, and physics data sets with varied task counts and types, it beat existing multi-task designs and single-task transformers on multi-task gain. The authors also release MultiTab-Bench, a synthetic generator that varies task count, correlation, and complexity for systematic study.

Catalog record

CREATORDimitrios Sinodinos, Jack Yi Wei, Narges Armanfard
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-09970

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MultiTab: A Scalable Foundation for Multitask Learning on Tabular Data Dimitrios Sinodinos, Jack Yi Wei, Narges Armanfard Abstract Tabular data is the most abundant data type in the world, powering systems in finance, healthcare, e‑commerce, and beyond. As tabular datasets grow and span multiple related targets, there is an increasing need to exploit shared task information for improved multitask generalization. Multitask learning (MTL) has emerged as a powerful way to improve generalization and efficiency, yet most existing work focuses narrowly on large‑scale recommendation systems, leaving its potential in broader tabular domains largely underexplored. Also, existing MTL approaches for tabular data predominantly rely on multi-layer perceptron-based backbones, which struggle to capture complex feature interactions and often fail to scale when data is abundant, a limitation that transformer architectures have overcome in other domains. Motivated by this, we introduce MultiTab-Net, the first multitask transformer architecture specifically designed for large tabular data. MultiTab-Net employs a novel multitask masked‑attention mechanism that dynamically models feature–feature dependencies while mitigating task competition. Through extensive experiments, we show that MultiTab-Net consistently achieves higher multitask gain than existing MTL architectures and single‑task transformers across diverse domains including large‑scale recommendation data, census‑like socioeconomic data, and physics datasets, spanning a wide range of task counts, task types, and feature modalities. In addition, we contribute MultiTab-Bench, a generalized multitask synthetic dataset generator that enables systematic evaluation of multitask dynamics by tuning task count, task correlations, and relative task complexity. Accepted for publication at AAAI 2026 Code — https://github.com/Armanfard-Lab/MultiTab Introduction Tabular data is a fundamental data format that supports a wide range of industries, including finance, healthcare, and e-commerce. Its structured layout of rows and columns enables efficient storage, querying, and manipulation, supporting critical applications from large-scale recommendation systems ( 22 ) to population studies based on survey data and medical diagnosis ( 5 ) . As tabular data becomes increasingly ubiquitous, advancements in data collection and digital infrastructure have led to massive, high-dimensional datasets with multiple related prediction targets. In healthcare, for instance, patient records containing age, BMI, and lab results can be used to predict both the presence of diabetes and the risk of high blood pressure. Similarly, on an online retail platform, it is desirable to not only predict if a user will click on a product, but also if they will add it to their cart and make a purchase ( 33 ) . Figure 1: Overview of MultiTab-Net, our proposed multitask transformer for tabular data. A sample of d features (categorical and/or numerical) and t task tokens are passed through an embedding network to generate embeddings of size e for each token. Task tokens are appended to the feature tokens, forming a combined input sequence, which is then processed by stacked encoder blocks. Each encoder block consists of inter-feature and inter-sample attention modules, along with feed-forward networks and residual connections. After N encoder blocks, the processed task tokens T^{\prime}=\{T^{\prime}_{1},\dots,T^{\prime}_{t}\} are passed through task-specific multilayer perceptrons (MLP) to generate the final task predictions T^{P}=\{T^{P}_{1},\dots,T^{P}_{t}\} . In such settings, multitask learning (MTL) ( 4 ) offers a way to leverage task correlations and improve generalization by training models on multiple tasks simultaneously. This shared learning enhances feature representations, mitigates overfitting, and acts as a form of regularization ( 35 ) . Beyond improving performance, MTL improves computational efficiency by enabling a single model to handle multiple tasks in parallel, reducing training and inference time ( 28 ) . In large-scale production environments, particularly recommendation systems, MTL has already proven effective in modeling related metrics such as click-through rate (CTR) and conversion rate (CVR) ( 23 ; 34 ; 33 ) . However, MTL for tabular data remains largely underexplored outside of recommendation systems and existing approaches typically rely on simple multi-layer perceptron (MLP) backbones ( 23 ; 34 ; 33 ) . These MLP-based backbones lack explicit architectural mechanisms for modeling complex inter-feature interactions and task-specific dependencies ( 7 ) , which can limit their scalability as datasets and task counts grow ( 25 ) . Transformer architectures offer a promising alternative. While MLPs implicitly learn feature interactions through dense layers, transformers can leverage self-attention to dynamically model the relationships between features and tasks ( 36 ) . In domains like NLP and vision, transformers have demonstrated their strength in capturing long-range dependencies and complex interactions, particularly when trained on large datasets ( 8 ; 10 ) . Specifically for tabular data, transformers can also capture sample-level dependencies through self-attention, modeling relationships across rows and achieving substantial performance gains ( 30 ) . This ability to model both inter-feature and inter-sample dependencies is particularly valuable for tabular datasets, where the relative importance of columns (features) and rows (samples) often varies across tasks. These advantages, combined with the increasing availability of large-scale, multi-target tabular datasets motivates the exploration of transformer-based architectures for MTL in tabular data. We introduce MultiTab-Net, a novel multitask transformer designed specifically for tabular data. It combines the generalization and efficiency benefits of multitask learning with transformers’ ability to model complex feature interactions via attention. A key innovation is our multitask masked attention framework, the first such mechanism applied to multitask transformers. While demonstrated on tabular data, this framework has broader potential in domains like computer vision and NLP, where managing task competition while capturing intricate feature interactions is equally critical. This mechanism prevents task interference in inter-feature learning ( 19 ) , leading to superior multitask gain, \Delta_{m} ( 24 ) , on large-scale datasets, outperforming traditional multitask MLPs and single-task transformers. Notably, the magnitude of these gains exceeds those typically reported in prior multitask tabular works ( 23 ; 34 ; 33 ) . In addition to public tabular datasets, synthetic data has become a standard approach for evaluating multitask learning performance and robustness to variations in task correlation ( 23 ; 34 ) . The existing method by ( 23 ) allows for tunable task correlations but is limited to two tasks and lacks a notion of relative task difficulty—both key factors in multitask learning dynamics ( 35 ) . These constraints limit its effectiveness in analyzing multitask behaviors. To address this, we introduce MultiTab-Bench, a novel synthetic multitask dataset generator that enables full tunability of pairwise task correlations and relative task difficulty for any number of tasks, providing a more comprehensive synthetic baseline for multitask evaluations in tabular settings. Overall, our contributions are as follows: • We introduce MultiTab-Net , the first multitask transformer architecture for tabular data, which outperforms existing MLP-based multitask models and single-task transformer models on widely used tabular benchmarks. • We propose a novel multitask masked attention method that mitigates the effects of task competition during inter-feature relationship learning. • We develop MultiTab-Bench , a new synthetic multitask tabular dataset generator that enables fine-grained control over pairwise task correlations and relative task difficulty for any number of tasks, allowing for evaluation on virtually any multitask learning setting. Related Works Deep Learning has transformed domains like vision and language ( 21 ; 8 ) , but has historically struggled to handle the heterogeneous feature types and irregular distributions common in tabular data. Recent studies now show that deep learning can also achieve competitive or superior performance on tabular benchmarks ( 25

Read the complete text