Task Scarcity and Label Leakage in Relational Transfer Learning

Francisco Galuppo Azevedo, Clarissa Lima Loures, Denis Oliveira Correa · 2026

Read Task Scarcity and Label Leakage in Relational Transfer Learning by Francisco Galuppo Azevedo, Clarissa Lima Loures, Denis Oliveira Correa (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29914.txt.

Introduction

Galuppo Azevedo and co-authors study why relational models transfer poorly across tasks. Databases offer few prediction targets, so training data stays scarce per task. Models then learn task-specific shortcuts that hurt transfer, a flaw they call label leakage. Their K-Space design pairs frozen table encoders with a light message-passing core. A gradient projection step strips label-predictive directions from updates. On RelBench, transfer rises by 0.145 AUROC on average. It often nearly matches single-task scores.

Catalog record

CREATORFrancisco Galuppo Azevedo, Clarissa Lima Loures, Denis Oliveira Correa
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19723
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29914

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Task Scarcity and Label Leakage in Relational Transfer Learning Francisco Galuppo Azevedo, Clarissa Lima Loures, Denis Oliveira Correa Abstract Training relational foundation models requires learning representations that transfer across tasks, yet available supervision is typically limited to a small number of prediction targets per database. This task scarcity causes learned representations to encode task-specific shortcuts that degrade transfer even within the same schema, a problem we call label leakage. We study this using K-Space, a modular architecture combining frozen pretrained tabular encoders with a lightweight message-passing core. To suppress leakage, we introduce a gradient projection method that removes label-predictive directions from representation updates. On RelBench, this improves within-dataset transfer by +0.145 AUROC on average, often recovering near single-task performance. Our results suggest that limited task diversity, not just limited data, constrains relational foundation models. 1 Introduction Relational databases remain the dominant substrate for operational data. Building foundation models for this setting requires representations that transfer across tasks. Yet a typical relational database offers only a handful of supervised prediction tasks. This task scarcity is a data bottleneck distinct from sample scarcity: even with millions of rows, the number of distinct objectives available to shape representations is small. When supervision comes from a single task, learned representations are free to encode any feature that predicts the label, including task-specific shortcuts that do not generalize. We call this label leakage: the representation absorbs label-predictive directions that help the training objective but hurt transfer. With abundant task diversity, conflicting gradients from different objectives would suppress such shortcuts. Under task scarcity, no such pressure exists. As we show empirically, within-dataset transfer can collapse to near-chance performance even when single-task accuracy is strong. We study this problem within the Relational Deep Learning (RDL) framework, which constructs heterogeneous graphs from multi-table databases and learns directly over that structure 3 ; 9 . Our testbed is K-Space , a modular RDL architecture with a convolutional message-passing core that has linear edge complexity. Row-level representations come from a frozen pretrained tabular encoder, and predictions are made by a frozen in-context learning head 7 . This modularity isolates the relational backbone as the component where leakage can occur. To suppress it, we introduce a gradient projection method: at each training step, we remove from the main loss gradient any component that aligns with an adversarial label-prediction gradient, computed per sample at the representation level. Contributions. • We empirically demonstrate that task scarcity causes label leakage in relational learning: within-dataset transfer often collapses to near-chance performance despite strong single-task accuracy. • We propose a sample-wise gradient projection method that suppresses label-predictive directions in the shared representation during training. • We show that this intervention improves within-dataset transfer (+0.145 AUROC on average) but does not help and can hurt cross-dataset transfer, suggesting different mechanisms underlie the two settings. 2 Related Work RDL and RelBench. Relational deep learning constructs heterogeneous graphs from multi-table databases and trains neural predictors over these structures 3 . RelBench provides standardized datasets and tasks for this setting, with multiple tasks per database 9 . This enables evaluation of within-dataset transfer. Relational transformers. Recent transformer-based approaches target strong single-task performance and transfer 1 ; 12 ; 8 . RelGT achieves the highest reported results on RelBench under supervised training and serves as our primary baseline. Relational Transformer takes a different approach, using masked prediction to create self-supervised tasks from unlabeled data, sidestepping task scarcity. We explore a complementary direction: mitigating label leakage when only supervised tasks are available. Pretrained tabular encoders and ICL heads. TabICL provides a scalable tabular ICL model whose embedding stages yield strong row representations 7 . TabPFN-style models illustrate that ICL heads can be effective predictors for tabular tasks, including variants that support regression 5 . Freezing both encoder and predictor isolates the relational backbone as the sole learnable component and the locus of potential label leakage. Scalable message passing. SMPNN shows that deep message passing can be stabilized using transformer-like residual blocks while preserving linear edge complexity 10 . K-Space adopts SMPNN as the relational building block and extends it to heterogeneous directed propagation with shared weights. A simple architecture allows transfer failures to be attributed to the representation rather than architectural mismatches. Adversarial methods for representation learning. Adversarial training can remove unwanted information from representations. Domain-adversarial neural networks (DANN) use gradient reversal to learn domain-invariant features 4 , but we found this unstable for removing task-specific label information. Instead, inspired by Muon 6 , we subtract the adversarial gradient’s component from the main gradient. 3 Architecture \figtree Input Projection \figtree Hetero Block \figtree N\times \figtree REV Block \figtree FWD Block \figtree + \figtree RoPE \figtree Output projection \figtree Layer Norm \figtree GCN \figtree Scale \figtree + \figtree Layer Norm \figtree Pointwise Feedforward \figtree Scale \figtree Modified SMPNN Block \figtree GCN \figtree Linear Projection \figtree GCNConv / GATv2 \figtree SiLU \figtree Pointwise FF \figtree Linear Projection \figtree SiLU \figtree Adversarial Head \figtree [TabICL] Predictor \figtree 25.78M aaa \figtree ❄ \figtree Label Prediction \figtree In-Context Label Prediction \figtree Time Features \figtree RWPE \figtree Table Features \figtree [TabICL] Encoder \figtree 1.28M aaa \figtree ❄ Figure 1: Schematic of the proposed architecture. A TabICL encoder (column embedder and row interactor) encodes table features; its output is concatenated with RWPE and time features and passed through an input projection and per-table-type RoPE into a stack of N Hetero (SMPNN) blocks with reversible (REV) and forward (FWD) paths. Zoomed views show the SMPNN block and its GCN and pointwise feedforward sub-blocks. An output projection feeds two heads: an adversarial MLP head and a TabICL predictor head. We now describe the K-Space architecture and our gradient projection implementation. It has three blocks: a frozen table encoder, a relational convolutional core, and an ICL predictor head, see Figure 1 . Frozen table encoder. Each node begins as a row in some table. We compute row embeddings using the frozen embedding stages of TabICL 7 . To these, we concatenate temporal features (index plus periodic encodings) and random walk positional encodings (RWPE). Unlike the original RWPE formulation 2 , which uses matrix exponentiation, we compute encodings via explicit random walks that cannot traverse future edges. The result is projected to the model width, with table type injected via RoPE 11 . Relational convolution. We build on SMPNN 10 . Each layer uses shared parameters across relation types and separates incoming from outgoing propagation. For each relation type r, we compute an update from incoming neighbors and an update from outgoing neighbors, aggregate across r, and apply a residual update. The full procedure is given in Algorithm 1 (Appendix A ). ICL predictor head We predict binary labels via a pretrained frozen ICL head: each mini-batch splits labeled instances into support (embedding, label) pairs and query embeddings. The head consumes support context and outputs query probabilities. Our experiments focus on classification due to the TabICL predictor interface 7 . Label-leakage suppression All reported K-Space results use the same architecture; we vary only whether training includes an adversarial label-prediction head. The motivation is to reduce direct label information in representations, which can hinder transfer when tasks change. Let \mathbf{z} denote the query representation after the GNN and before the ICL head. We train

Read the complete text