TabPATE: Differentially Private Tabular In-Context Learning Without Public Data

Dariush Wahdany, Matthew Jagielski, Jesse C. Cresswell, Adam Dziedzic, Franziska Boenisch · 2026

Read TabPATE: Differentially Private Tabular In-Context Learning Without Public Data by Dariush Wahdany, Matthew Jagielski, Jesse C. Cresswell, Adam Dziedzic, Franziska Boenisch (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31474.txt.

Introduction

Wahdany and four colleagues present TabPATE, a way to use private tables inside a foundation model's context without leaking them. Tabular in-context learning predicts from a small labeled dataset placed in the prompt, and in health or finance those rows are sensitive. The team first show that membership inference attacks, even basic ones, can tell whether a row was in the context.

TabPATE splits the private context across teacher models in the PATE style and aggregates their labels privately on synthetic queries. The labeled synthetic set then serves as the student's context, so real rows never reach the final model. Queries can be built from feature ranges alone, with no public data. Across benchmarks, utility stays competitive while membership inference falls to near-random success.

Catalog record

CREATORDariush Wahdany, Matthew Jagielski, Jesse C. Cresswell, Adam Dziedzic, Franziska Boenisch
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31474

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TabPATE: Differentially Private Tabular In-Context Learning Without Public Data Dariush Wahdany, Matthew Jagielski, Jesse C. Cresswell, Adam Dziedzic, Franziska Boenisch Abstract Tabular foundation models enable accurate in-context learning (ICL) from small labeled datasets, but the private records placed in context can leak through model predictions. We first show that even basic membership inference attacks succeed against tabular ICL, motivating formal privacy protection. We then introduce TabPATE, a differentially private PATE-style defense for tabular ICL that does not require public in-distribution data. TabPATE partitions the private context across teacher models, privately aggregates their labels on synthetic tabular queries, and releases the resulting labeled queries as a student context. Because tabular features are bounded and relatively low-dimensional, useful queries can be generated from feature ranges alone or from lightly privatized marginals. Across tabular benchmarks, TabPATE preserves competitive utility while reducing membership inference to near-random success, providing a practical path to private tabular ICL without public data. 1 Introduction \Acp TFM enable accurate predictions on structured data via ICL ( ICL ): a small labeled dataset is placed in the model context and used to predict labels for new queries, without task-specific training ( 14 ; 15 ) . This makes TFM appealing in domains such as healthcare and finance, where tabular data is common and labeled data is often scarce ( 31 ; 17 ) . However, the context records may contain private information. We first verify that this concern is not merely hypothetical. In ICL , membership inference asks whether a target record was included in the private demonstration context. Adapting standard MIA ( 27 ; 6 ) to tabular ICL , we find that prediction behavior reveals non-trivial information about context membership. Even a passive grey-box adversary obtains 9.9\% TPR ( TPR ) at 1\% FPR ( FPR ), while stronger active attacks can reach substantially higher leakage. These results motivate mechanisms that protect the in-context data itself, rather than only the pretraining data of the foundation model. DP ( DP ) ( 11 ) provides a formal way to limit the influence of any individual private record. Yet standard DP training methods are not directly suited to tabular ICL . DPSGD ( DPSGD ) ( 1 ) requires gradients and parameter updates, whereas TFM are used as frozen inference-time models. PATE ( PATE ) ( 23 ; 24 ) , in contrast, is naturally compatible with ICL as private data can be split across teacher contexts, whose predictions are aggregated with calibrated noise. This idea underlies PromptPATE ( 10 ) in the language domain. Like all prior PATE-style methods, PromptPATE requires unlabeled public data from the target distribution for knowledge transfer. In sensitive tabular domains, such public in-distribution data is often unavailable. We introduce TabPATE , a central-DP framework for tabular ICL that removes this public-data requirement. TabPATE partitions the private context across teacher TFM , queries them on synthetic tabular records, privately aggregates their votes using Confident-GNMax algorithm ( 24 ) , and releases the resulting labeled synthetic records as a student context for privacy-preserving deployment. The key observation is that tabular domains differ from images and text: feature spaces are bounded, typed, and comparatively low-dimensional. Thus, useful teacher queries can be generated without public examples , either from a data-independent bounds-based prior or from a small DP release of marginal statistics. This yields a PATE-style defense for tabular ICL with no public in-distribution data. We evaluate our TabPATE on standard tabular benchmarks and compare against non-private ICL , PromptPATE, query-time aggregation, and DP synthetic-data baselines. TabPATE preserves competitive utility at moderate privacy budgets while reducing MIA success close to random guessing. Our results suggest that public-data-free PATE-style knowledge transfer is a practical route to private deployment of TFM . In summary, we make the following contributions: 1. We adapt membership inference to tabular ICL and show that private context records leak through model predictions, motivating formal protection. 2. We propose TabPATE, a central-DP PATE-style framework for tabular ICL that replaces public in-distribution transfer data with synthetic tabular queries. 3. We evaluate TabPATE across tabular benchmarks, showing competitive utility at moderate privacy budgets and near-random membership inference success. 2 Related Work Tabular foundation models. \Acp TFM perform prediction on structured data via transformer-based ICL ( ICL ) ( 32 ; 4 ) . Given labeled demonstrations in the context, TFMs predict labels for new queries without updating model parameters. TabPFN ( 14 ; 15 ) is a prominent example, pretrained on synthetic tasks and competitive with strong baselines. Recent models such as TabDPT ( 18 ) and TabICL ( 25 ; 26 ) further improve tabular ICL performance, with benchmarks tracking rapid progress ( 12 ) , and applications to adjacent fields ( 2 ; 28 ) . We study the privacy of this inference-time learning paradigm, where sensitive records are placed directly in the model context. Membership inference for ICL. \Acp MIA ( 27 ) are the standard empirical tool for measuring whether individual records influence model behavior. Modern attacks include calibrated likelihood-ratio attacks such as LiRA ( 6 ) , label-only perturbation attacks ( 8 ) , and poisoning-based amplifications such as Truth Serum ( 30 ) . For text ICL , Brainwash-style attacks show that context examples can leak through resistance to adversarial relabeling ( 33 ) . We adapt these attack ideas to tabular ICL to motivate the need for formal protection. Differential privacy and PATE. DP ( 11 ) bounds the effect of any single private record on a released computation. DPSGD ( 1 ) achieves this during training via gradient clipping and noise, but is not directly applicable to frozen ICL models. PATE ( 23 ; 24 ) instead partitions private data among teachers and transfers knowledge through noisy vote aggregation, making it a natural fit for ICL . PromptPATE ( 10 ) applies this idea to language-model prompting, but like standard PATE, relies on public in-distribution data for teacher queries. Other DP-ICL methods either answer each query privately at inference time ( 34 ) or target autoregressive text generation ( 29 ) . TabPATE follows the PATE-style student-release approach, but removes the public-data requirement by generating synthetic tabular queries. DP tabular synthetic data. DP synthetic data methods such as AIM ( 20 ) and MST ( 19 ) privately estimate low-dimensional marginals and post-process them with Private-PGM ( 21 ) to generate synthetic tables. We use such methods as baselines, but TabPATE differs in goal: it does not aim to release a synthetic version of the private dataset, but only to generate useful unlabeled queries for private PATE-style knowledge transfer. 3 Membership Inference in Tabular ICL Before introducing TabPATE, we verify that private in-context records can leak through the predictions of a TFM . In standard training-based MIA , membership asks whether a record was used for training ( 27 ) . In tabular ICL , the pretrained model is fixed; MIAs instead ask whether a target record was included in the private demonstration context used at inference time. We consider adversaries along two axes: output access , where the model returns either labels only (black-box) or prediction probabilities (grey-box), and context manipulation , where the adversary is either passive or

Read the complete text