MIDST Challenge at SaTML 2025: Membership Inference over Diffusion-models-based Synthetic Tabular data

Masoumeh Shafieinejad 1 , Xi He 1,2 , Mahshid Alinoori 1 , John Jewell 1 , Sana Ayromlou 3 , Wei Pang 2 , Veronica Chatrath 4 , Gauri Sharma 5 , Deval Pandya 1 · 2026

Read MIDST Challenge at SaTML 2025: Membership Inference over Diffusion-models-based Synthetic Tabular data by Masoumeh Shafieinejad 1 , Xi He 1,2 , Mahshid Alinoori 1 , John Jewell 1 , Sana Ayromlou 3 , Wei Pang 2 , Veronica Chatrath 4 , Gauri Sharma 5 , Deval Pandya 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19185.txt.

Introduction

Shafieinejad and co-authors report on MIDST, a challenge that tested the privacy of synthetic tables made by diffusion models. Such data should keep the stats of real records while hiding who was in them. Teams built black-box and white-box membership attacks for single tables and linked multi-table cases. The attacks try to tell if a known record helped train the model. The event gave a shared way to grade privacy claims. Its code and tasks are public on GitHub.

Catalog record

CREATORMasoumeh Shafieinejad 1 , Xi He 1,2 , Mahshid Alinoori 1 , John Jewell 1 , Sana Ayromlou 3 , Wei Pang 2 , Veronica Chatrath 4 , Gauri Sharma 5 , Deval Pandya 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16322
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19185

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MIDST Challenge at SaTML 2025: Membership Inference over Diffusion-models-based Synthetic Tabular data Masoumeh Shafieinejad 1 , Xi He 1,2 , Mahshid Alinoori 1 , John Jewell 1 , Sana Ayromlou 3 , Wei Pang 2 , Veronica Chatrath 4 , Gauri Sharma 5 , Deval Pandya 1 Abstract Synthetic data is often perceived as a silver-bullet solution to data anonymization and privacy-preserving data publishing. Drawn from generative models like diffusion models, synthetic data is expected to preserve the statistical properties of the original dataset while remaining resilient to privacy attacks. Recent developments of diffusion models have been effective on a wide range of data types, but their privacy resilience, particularly for tabular formats, remains largely unexplored. MIDST challenge sought a quantitative evaluation of the privacy gain of synthetic tabular data generated by diffusion models, with a specific focus on its resistance to membership inference attacks (MIAs). Given the heterogeneity and complexity of tabular data, multiple target models were explored for MIAs, including diffusion models for single tables of mixed data types and multi-relational tables with interconnected constraints. MIDST inspired the development of novel black-box and white-box MIAs tailored to these target diffusion models as a key outcome, enabling a comprehensive evaluation of their privacy efficacy. The MIDST GitHub repository is available at: https://github.com/VectorInstitute/MIDST [email protected], [email protected], [email protected], [email protected], [email protected] I Introduction Privacy regulations across the globe — European General Data Protection Regulation (GDPR), the Canadian Personal Information Protection and Electronic Documents Act (PIPEDA), California’s Consumer Privacy Act (CCPA), etc — call for data anonymization as a main privacy principle. Industries are exploring privacy-preserving technologies to comply. JP Morgan has published an extensive report potluru2024synthetic stating “for a highly regulated finance industry, synthetic data is a potential approach for dealing with issues related to privacy, fairness, and explainability”. Similarly, the US Department of Health and Human Services regards synthetic data as a powerful tool that can address main challenges in making health care data accessible Gonzales2023Synthetic . Vector Institute, a national AI institute in Canada, organized a boot camp for industry sponsors to promote diffusion models for synthesizing tabular data vector2025DMbootcamp . However, without effective measures against privacy attacks, true anonymity may not be achieved. Vector Institute, organized a membership inference attack challenge on synthetic tabular data generated by diffusion models, named MIDST challenge, as part of the IEEE SaTML 2025 conference. This competition is a contribution to the broader, real-world challenge of translating industry and regulatory risk assessments into technical terms. I-A Novelty The usage of membership inference attacks for anonymity measurement is not new. However, their application on complex tabular data generated by diffusion models had not yet been addressed by either the research community or the industry prior to MIDST. MIDST also extended this evaluation to multi-table data synthesis, a popular application in industry, which had not been addressed for evaluation in research before, even outside the diffusion model context. A relevant membership inference competition was held by Microsoft in SaTML 2023, MICO project microsoft2023mico , evaluating the effectiveness of differentially private model training as a mitigation against white-box membership inference attacks. However, MIDST focuses on complex tabular data and evaluates the privacy efficacy or limitations of synthetic data generators. II Background II-A Tabular Data Synthesis Synthetic data has attracted significant interest for its ability to tackle key challenges in accessing high-quality training datasets. These challenges include: i) privacy assefa2020generating ; hernandez2022synthetic , ii) bias and fairness Breugel2023Beyond , and iii) data scarcity fonseca2023tabular ; zheng2022diffusion . The interest in synthetic data has extended to various commercial settings, notably in healthcare Gonzales2023Synthetic and finance potluru2024synthetic sectors. The synthesis of tabular data, among all data modalities, is a critical task with approximately 79% of data scientists working with it on a daily basis Breugel2023Can . II-B Diffusion Models Diffusion models have emerged as powerful tools for data synthesis, demonstrating remarkable success in various domains rombach2022high . These models are particularly noted for their strong capabilities in controlled (conditioned) generation. They have been used for generating synthetic data in both unconditional settings, for single tables, kotelnikov2023tabddpm ; zhang2023mixed ; Lee2023CoDi ; kim2022stasy and pang2024clavaddpmm ; Liu2024Controllable conditioned setting for multiple interconnected tables. II-C Membership Inference Attacks (MIAs) In tabular synthesis literature, the generative model is commonly evaluated by a basic distance to the closest record (DCR) metric for its privacy protection. However, the generated synthetic data indicates its privacy deficiencies when assessed by stronger privacy measurements such as membership inference attacks Stadler2022Synthetic ; giomi2022unified . The evaluation of diffusion models by membership inference attacks was largely limited to computer vision Breugel2023Membership ; Duan2023AreDiff ; Carlini2023Extracting , prior to the MIDST competition. With the success of diffusion models in generating tabular synthetic data, MIDST highlighted the need to evaluate the models with proper privacy metrics. III MIDST Challenge Design The generative models are developed on the training data set to generate synthetic data. They are expected to learn the statistics without memorizing the individual data. To evaluate this promise, membership inference attacks assess whether the model distinguishes between the training data set and a holdout data set , both are derived from the same, larger data set. For each of the four tasks, a set of models were trained on different splits of a public dataset. For each of these models, m challenge points were provided; exactly half of which are members (i.e., used to train the model) and half are non-members (i.e., from the holdout set; they come from the same public dataset as the training set, but were not used to train the model). The goal of the participants is to determine which challenge points are members and which are non-members. III-A Challenge Tracks MIDST challenge was composed of four different tracks, each associated with a separate category. The categories are defined based on the access to the generative models and the type of the tabular data as follows: 1. Access to the models: black-box, Data: single table 2. Access to the models: white-box, Data: single table 3. Access to the models: black-box, Data: multi-table 4. Access to the models: white-box, Data: multi-table In white-box attacks, the participants had access to the models and their generated synthetic output. Training sets for these models are selected from a public dataset. In black-box attacks, the access to the same information as the white-box attack was granted, except for the models. III-B Models and Datasets Vector Institute held a boot camp on diffusion-model-based tabular synthesis used for a variety of applications for industry stakeholders on September 3-5, 2024 vector2025DMbootcamp . The plug-and-play diffusion-model-based reference implementations include: TabDDPM (single table) kotelnikov2023tabddpm , TabSyn (single table) zhang2023mixed , and ClavaDDPM (multi-table) pang2024clavaddpmm . The repository of the models and the utilized datasets are publicly available through the link: https://github.com/VectorInstitute/MIDSTModels . The same models were included in the MIDST challenge for privacy evaluation. To facilitate the participation of all interested researchers regardless of their computing capabilities, MIDST also provided a set of 30 shadow models for each of these three models. The shadow models were the same for black-box and white-box tasks. The participants were free to choose these shadow models and/or generate their own if needed in developing their MIAs. These models were each trained

Read the complete text