Probing Memorization of Tabular In-Context Learning

Francesco Capano, Jonas Böhler · 2026

Read Probing Memorization of Tabular In-Context Learning by Francesco Capano, Jonas Böhler (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31208.txt.

Introduction

Capano and Böhler probe whether large tabular models secretly memorize their training rows. Such models use in-context learning, reading example rows in the prompt to predict on new tables. Their IclMem framework strips valid patterns from the context, so any correct answer must come from memory stored in the model weights.

A controlled fine-tuning setup sets membership ground truth and corrects for pitfalls like distribution shift and base-rate errors. The probe finds moderate memorization in eight of ten tasks, strongest for binary and low-cardinality data. Yet signals largely vanish under realistic training, appearing mainly after long single-task fine-tuning on fixed samples. The authors discuss safeguards for sensitive tables where even narrow memorization could leak private rows.

Catalog record

CREATORFrancesco Capano, Jonas Böhler
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31208

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probing Memorization of Tabular In-Context Learning Francesco Capano, Jonas Böhler Abstract Large tabular models (LTMs), i.e., tabular foundation models leveraging in-context learning (ICL), achieve state-of-the-art performance on tabular tasks. While LLMs are known to unintentionally memorize training data, the memorization dynamics of LTMs remain largely unexplored. We investigate the potential for parametric memorization in tabular ICL. We introduce IclMem , a probing framework designed to separate context-based predictions from parametric memorization. Our zero-information multiple-choice context strips away valid contextual patterns to force the model to fall back on its parametric memory. Our controlled fine-tuning setup establishes membership ground truth and accounts for common pitfalls, e.g., distribution shift, feature contamination, base-rate fallacy, and the pre-trained base model acts as reference to calibrate for sample difficulty. Our controlled evaluation on a leading real-world-trained LTM detects moderate memorization signals in 8 out of 10 tasks ( AUC up to 0.67 0.67 and TPR at 1 % 1\% FPR > 0.1 >0.1 ). Notably, memorization signals are strongest for low-cardinality and binary tasks. However, they largely vanish under realistic training conditions. Our findings show LTM memorization signals under specific circumstances (single-task fine-tuning with fixed samples across many epochs and small query size). To protect sensitive data, appropriate measures must be taken, which we discuss. 1 Introduction Tabular data dominate enterprise applications and domains such as healthcare and finance ( 12 ) . Large tabular models (LTMs) ( 39 ; 22 ; 37 ) , i.e., tabular foundation models, achieve state-of-the-art performance for tabular predictions by leveraging in-context learning (ICL) to perform zero-shot predictions on unseen tasks ( 5 ) . At inference, the model receives a context (e.g., historical example rows) alongside a query row with a missing target field to be predicted. While LTMs are typically pre-trained on synthetic data ( 22 ; 33 ) , recent models incorporate real-world tabular corpora ( 37 ; 16 ; 28 ) , e.g., by fine-tuning on diverse real-world task mixtures to improve performance ( 19 ) . While LLMs are known to unintentionally memorize training data ( 10 ; 21 ; 38 ) , the memorization dynamics of LTMs remain largely unexplored. Rigorous memorization probing is crucial to inform potential mitigations for sensitive fine-tuning data, such as differential privacy and adaptation of data pre-processing and training regimes. Problem Setting. Traditional LLM ICL x_{1} y_{1} \cdots x_{n} y_{n} Context C Model Provider q_{x} ? Query q User Goal: Infer if q\in C LLM Prediction \hat{y} LTM ICL x_{1} y_{1} \cdots x_{n} y_{n} Context C q_{x} ? Query q User Goal: Infer if q\in D LTM M_{\theta} Fine-tuned on D Prediction \hat{y} Figure 1 : Memorization probing for LLM vs LTM. Left: In LLMs with ICL the model provider supplies a sensitive context C , which the user attempts to extract via user-controlled queries. Right: In LTMs, the user controls both the context C and the query q . In IclMem the context is used to probe whether q was memorized from the sensitive fine-tuning data D . In the ICL paradigm, a model predicts a missing target value y_{q} given query’s input features x_{q} , and a context C=\{(x_{1},y_{1}),\dots,(x_{n},y_{n})\} of examples. With LTMs, the user provides both the context C and the query q , as shown in Fig. 1 . We assume an LTM M_{\Theta} is fine-tuned on sensitive D by a data owner who provides access to the fine-tuned LTM to users. This setting opens up the possibility to manipulate C arbitrarily, to probe differences in model behavior on context-query pairs derived from data included in or excluded from D . In contrast, existing works on ICL privacy primarily assume the model provider defines a sensitive C , which an attacker attempts to extract via membership inference attacks (MIA) by manipulating q ( 41 ; 14 ) . MIAs ( 35 ) aim to infer whether a specific record was part of the training data, and transferring MIAs to LTMs presents unique challenges. Unlike LLMs, which can generate arbitrary text on a vast output space to extract memorized tokens ( 9 ) , LTMs are encoder-only architectures that strictly constrain predictions to the pre-defined candidate labels provided in C ( 37 ; 22 ) . More importantly, while masked-language modeling can induce memorization ( 20 ) , LTMs operate as Prior-Data Fitted Networks (PFNs) ( 31 ) . They are trained to infer structural connections within the context rather than memorize explicit targets ( 31 ) . This raises a fundamental, yet unanswered question: What memorization profile do LTMs, inherently trained to perform ICL, have? To investigate memorization capabilities of LTMs, we present IclMem and apply MIA. Specifically, we test membership of a candidate set of query rows by manipulating C as an active probing mechanism (Fig. 1 ). We hypothesize that by carefully modifying C to eliminate discriminative information, it disrupts the model’s context-derived prediction capabilities. Forced to abandon in-context reasoning, the model falls back on its memorized parametric knowledge. For member rows in D , we expect predictions to exhibit higher confidence and remain robust to context manipulations ( 11 ; 42 ) . We identify specific circumstances for memorization in LTMs (Sec. 4.2 ) and discuss protection measures (Sec. 6 ). 2 Related Work While LTMs ( 22 ; 33 ; 37 ; 16 ) rapidly advance, investigations into their privacy properties remain limited ( 32 ) . Existing LTM privacy literature primarily targets test-time evasion and adversarial robustness rather than memorization ( 36 ; 13 ; 3 ) . Conversely, ICL privacy research largely focuses on LLMs, probing either the leakage of hidden context examples ( 41 ; 14 ) or table memorization within text corpora ( 17 ; 4 ) . To assess memorization, state-of-the-art MIAs typically rely on shadow models ( 6 ; 44 ) . However, training shadow models for large models is computationally prohibitive ( 21 ) and yields unreliable false-positive bounds when the pre-training distribution is unknown ( 45 ) . Towards privacy assessment of LTMs, we adapt techniques from robustness-assessing MIA ( 11 ) and attribute inference (AIA) ( 23 ; 2 ; 34 ) . While AIA aims to infer an unknown sensitive value given a partial record, our goal is inferring membership of a record. To assess memorization in LTMs, we adapt AIA’s multiple-choice testing and label randomization and combine zero-information context manipulations with difficulty calibration via the pre-trained base model ( 40 ) . 3 Contributions We provide the first systematic assessment of memorization in LTMs. Our contributions are: • We introduce IclMem , a probing framework designed to separate context-based predictions from parametric memorization. Our zero-information multiple-choice context strips away contextual patterns forcing the model to fall back on parametric memorization. • We avoid common MIA false positives by accounting for distribution shifts (via data pre-processing), feature contamination (via data deduplication), base-rate confounding (via label randomization), and intrinsic low-complexity samples (via difficulty calibration). • We perform an extensive evaluation on LTM ConTextTab across 1,128 fine-tuning configurations on 10 diverse classification and regression tasks from CARTE. • We detect moderate memorization in 8 of the 10 tasks (with \text{AUC}>0.5 and \text{TPR}_{@r\%}>0.1 for r\in\{1,10\} ) in our controlled setup with single-task fine-tuning

Read the complete text