LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains

Ibna Kowsar, Kazi F. Akhter, Manar D. Samad · 2025

Read LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains by Ibna Kowsar, Kazi F. Akhter, Manar D. Samad (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06161.txt.

Introduction

Ibna Kowsar, Kazi Akhter, and Manar Samad address a stubborn gap in transfer learning: tables from different fields rarely share the same columns. Large language models hold broad knowledge that could help, but plain text prompts and in-context learning stall on mixed table data. Their method, called LATTLE, moves knowledge through the model's attention weights instead.

The attention transplant lets learning cross between tables with no shared features, no prompt crafting, and no huge pretrained model. Tests used ten pairs of unrelated source and target datasets against twelve baseline methods. LATTLE beat traditional models and leading deep table architectures, even ones trained on far larger collections of tables.

Catalog record

CREATORIbna Kowsar, Kazi F. Akhter, Manar D. Samad
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06161

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LLM Attention Transplant for Transfer Learning of Tabular Data Across Disparate Domains Ibna Kowsar, Kazi F. Akhter, Manar D. Samad Abstract Transfer learning of tabular data is non-trivial due to heterogeneity in the feature space across disparate domains. The limited success of traditional deep learning in tabular knowledge transfer can be advanced by leveraging large language models (LLMs). However, the efficacy of LLMs often stagnates for mixed data types structured in tables due to the limitations of text prompts and in-context learning. We propose a lightweight transfer learning framework that fine-tunes an LLM using source tabular data and transplants the LLM’s selective k ​ e ​ y key and v ​ a ​ l ​ u ​ e value projection weights into a gated feature tokenized transformer (gFTT) built for tabular data. The gFTT model with cross-domain attention is fine-tuned using target tabular data for transfer learning, eliminating the need for shared features, LLM prompt engineering, and large-scale pretrained models. Our experiments using ten pairs of source-target data sets and 12 baselines demonstrate the superiority of the proposed LLM-attention transplant for transfer learning (LATTLE) method over traditional ML models, state-of-the-art deep tabular architectures, and transfer learning models trained on thousands to billions of tabular samples. The proposed attention transfer demonstrates an effective solution to learning relationships between data tables using an LLM in a low-resource learning environment. The source code of the proposed method is shared in 1 1 1 https://anonymous.4open.science/r/LATTLE—LLM-Tabular-Transfer-Learning-667E K eywords Tabular Data, Transfer Learning, Large Language Models, Cross-attention, Cross-domain. 1 Introduction Tabular data sets are structured in rows of samples and columns of mixed-type features, which are ubiquitous in spreadsheets and relational databases in countless domains. Despite widespread applications, tabular data have not attracted much attention in artificial intelligence (AI) research compared to image and text data. The recent AI revolution has ushered in a new era through deep learning (DL) of image and text, introducing large language models (LLMs), vision transformers (ViTs), and vision language models (VLMs). In contrast, a survey on deep learning of tabular data [ 3 ] reports that tabular data still rely on traditional machine learning (ML), particularly XGBoost [ 5 ] and LightGBM [ 18 ] , due to their superior performance over deep learning methods. Similar findings dissuade practitioners from reaping the benefits of deep representation and transfer learning for tabular data sets. Deep learning of tabular data is challenging for three reasons. First, the inductive bias required for robust representation learning of tabular data with mixed data types is not as developed as that of image and text data. Tabular data learning with mixed data types still relies on feature engineering with traditional ML. Second, many application domains have tabular data with limited samples for deep learning, whereas tabular data augmentation is not a trivial task [ 19 , 22 ] . Third, learning the relationship between tabular data sets of different domains is nontrivial due to the lack of transferable patterns or context in deep representations, similar to image and text data. Therefore, deep representation learning of shared knowledge and transferable context can pave the way for robust transfer learning of tabular data sets. The recent revolution in AI has demonstrated the versatility of large language models (LLMs) beyond text generation, even with minimal or no training samples. LLMs, as large reservoirs of knowledge, can perform zero- and few-shot learning of tabular data [ 11 , 15 , 12 ] . However, zero- and few-shot learning approaches perform poorly compared to traditional ML [ 25 ] . An efficient transfer learning method for tabular data can take full advantage of the knowledge base of LLM foundation models. This paper presents a novel cross-attention learning framework using LLMs that enables transfer learning between tabular data of different domains. The cross-attention between an LLM fine-tuned on the source data and a gated Feature Tokenizer Transformer (gFTT) is established by transplanting the frozen LLM weights into the gFTT for downstream classification of target data. The organization of the paper is as follows. Section 1.1 presents the latest deep learning and LLM approaches proposed in the context of tabular data and transfer learning. Section 2 discusses the preliminaries on attention-based computing and the proposed LLM attention transplant framework for transfer learning. Section 3 discusses the tabular data sets, the experimental setup and scenarios, and the evaluation method. Section 4 presents the transfer learning performance of the proposed method and compares it with other leading baselines for learning tabular data. Section 5 summarizes the key findings, provides insight into the results, and outlines the limitations. The paper concludes in Section 6 . 1.1 Literature review and contributions Traditional machine learning (ML) methods still have strong dominance in tabular data application domains due to their effectiveness in handling features of mixed data types [ 10 ] . Gradient-boosted decision trees, such as XGBoost [ 5 ] , perform well for structured tabular data, particularly when the data set is small. Recent deep learning methods, including TabNet [ 2 ] , Feature Tokenizer Transformer (FTT) [ 9 ] , and Gated Feature Tokenizer Transformer (gFTT) [ 30 ] , have been developed specifically to learn from tabular data. These methods demonstrate the promising performance of self-attention mechanisms, outperforming traditional ML methods in numerous instances with large sample sizes. TabPFN [ 13 ] proposes a tabular foundation model for small to medium-sized data sets using a probabilistic neural network approach. However, the model requires a vast amount of synthetic data during pretraining. A large pretrained model is often used in transfer learning when the sample size is too small for effective deep learning. In transfer learning, a model is first pretrained using a source data set with a large sample size, which is then fine-tuned using a target data set with limited samples. Several transfer learning methods have recently been proposed for tabular data sets. TransTab [ 30 ] is a state-of-the-art method that requires common features between the source and target data sets to align the data representations for knowledge transfer. In practice, two tabular data sets from different domains rarely share common features. Levin et al. [ 20 ] propose a similar transfer learning method between tabular data sets of the same domain with shared features. Ye et al. have introduced the cross-table mask modeling (CM2) framework [ 33 ] to learn generalizable data representations in heterogeneous feature spaces and tabular data domains, thus enhancing adaptation during fine-tuning. Similar to the CM2 method, XTab [ 36 ] pretrains an FTT model using tabular data sets from diverse domains. However, their downstream classification through fine-tuning often performs poorly compared to decision tree models (e.g., XGBoost [ 5 ] ). Therefore, existing methods struggle to optimally perform transfer learning for tabular data, particularly when the source and target schemas differ significantly. Furthermore, the realistic but challenging transfer learning between tabular data sets without common features has not been well studied. Figure 1: The proposed LLM attention transplant for transfer learning (LATTLE) framework. The attention-related weights are transplanted from a lightweight LLM (DistilGPT2) fine-tuned by source data to a Gated Feature Tokenizer Transformer (gFTT) to be fine-tuned using target data. Emb. = Embedding, Cat. = categorical and

Read the complete text