Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets

Nitish Nagesh, Mahdi Bagheri, Arshia Harish Puthran, Pengbao Zhou, Muhjaazee Love, Aadi Sharma, Ian Harris, Amir M. Rahmani · 2026

Read Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets by Nitish Nagesh, Mahdi Bagheri, Arshia Harish Puthran, Pengbao Zhou, Muhjaazee Love, Aadi Sharma, Ian Harris, Amir M. Rahmani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17758.txt.

Introduction

Nitish Nagesh and colleagues present Memisis, a tool that builds and judges synthetic health tables without exposing real patient records. It strings together existing synthesis libraries, language models, and evaluation measures. Users can set size, training length, and row counts, or describe goals in plain language for an agent mode to run. In a demo on an open schizophrenia dataset, the team compares six synthesizers spanning GANs, VAEs, diffusion models, and normalizing flows across privacy, usefulness, and fairness.

Catalog record

CREATORNitish Nagesh, Mahdi Bagheri, Arshia Harish Puthran, Pengbao Zhou, Muhjaazee Love, Aadi Sharma, Ian Harris, Amir M. Rahmani
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11678
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17758

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Memisis: Orchestrating and Evaluating Synthetic Data for Tabular Health Datasets Nitish Nagesh, Mahdi Bagheri, Arshia Harish Puthran, Pengbao Zhou, Muhjaazee Love, Aadi Sharma, Ian Harris, Amir M. Rahmani Abstract Synthetic data is widely used in healthcare to create datasets that are similar to original data but without the privacy concerns. Generating and evaluating synthetic data across privacy, utility and fairness is crucial for facilitating high quality data availability for downstream prediction tasks and clinical decision making. We present Memisis, a tool that orchestrates and evaluates synthetic data by leveraging existing synthetic data tools, the power of large language models and state-of-the-art evaluation metrics. Our tool creates a unified workflow for data generation, validation and evaluation. Users have control over the training size, training epochs and the number of synthetic rows to sample. Instead of knobs to tune synthetic data, the interactive agent allows users to specify their synthetic data generation goals and the tool will orchestrate the workflow by leveraging existing tools while performing the requisite evaluation. For the demo, we use an open source schizophrenia dataset with protected attributes related to race and gender, three different synthesizers and a local language model to orchestrate the workflow. We observe that CTGAN, TVAE and GaussianCopula have comparable performance across fairness and utility metrics. The workflow allows users flexibility and control over the data generation and evaluation process. Keywords: Synthetic Data, Fairness, Utility, Tabular Health Data 1 Introduction Generating synthetic data serves as a replacement to the original data especially when there are sensitive features, as is often the case with healthcare, where privacy preservation is key [ 9 ] . Architecting synthetic data generation frameworks contributes to improving access to high quality data that can be used in downstream applications for training and evaluation. Current synthetic data generators work well for tabular settings. Existing platforms range from generic synthetic data generators [ 11 ] to LLM-based applications [ 1 , 18 ] to cohort-specific simulators [ 19 ] to AI partners for data access and augmentation [ 4 ] . Our prior work [ 14 , 13 ] showcases how LLMs are used as synthetic data generators while evaluating fairness and utility metrics across a suite of open-source and restricted access datasets. However, there is a push to leverage agentic workflows due to reasoning, orchestration and tool calling capabilities [ 10 , 17 ] . We posit that agentic workflows for synthetic data in healthcare need to balance fairness and utility trade-offs [ 6 , 12 ] allowing a multi-faceted approach. To this end, we present, Memisis , an interactive agent that allows users to generate synthetic data using natural language. Our proposed method encompasses schema detection, synthesizer training, sampling and multi-metric evaluation giving flexibility to users for nuanced data synthesis and evaluation. 2 Method The tool coordinates with different generative models in the backend, plans and orchestrates analysis, presents the requisite data and shares reports. Workflow. We leverage existing synthesizers such as GANs, VAEs and GaussianCopula by connecting to the Synthetic Data Vault (SDV) [ 16 ] . A raw dataset including metadata is given as input. The generators use samples of the original data and are trained for a certain number of epochs, using a subset of the data. Using TSTR (Train Synthetic Test Real) paradigm, the validation is performed for evaluating distribution of data, which is a sign of utility. Going one step further, we evaluate fairness metrics, to investigate balance in the dataset. In the current version, the user exercises control through the generation knobs, as implemented in current data generators. Going beyond, the Memisis agent mode allows users agency to generate synthetic data through natural language while taking into consideration information about protected attributes. The agent orchestration is via a LangGraph ReACT (Reasoning and Acting) framework where a generator and evaluator coordinate to determine optimum generation mechanism based on designated distribution and fairness thresholds. In the multi-agent configuration, a supervisor partitions a generator subgraph from an evaluator subgraph, ensuring that scoring cannot silently influence synthesis. After evaluation, the supervisor compares outputs against parity thresholds and either routes back to the generator with a targeted refinement instruction or surfaces results to the user. Deployment stack. The architecture comprises of a python-based server deployed on the Streamlit cloud connected to GitHub. The chat model used is Llama3.2 (connected locally through Ollama). An NVIDIA RTX 3090 GPU is used for training the synthesizers. Evaluation and composite scoring. The orchestrator refines the responses based on user inputs, preferences and performance. The generated synthetic data is evaluated for utility using SDMetrics, a sister repository of SDV [ 5 ] . We evaluate fairness through false positive rates as illustrated in Fairlearn [ 2 ] . To jointly capture fairness–utility trade-offs, we devise a composite score that combines an established distributional fidelity score with an established fairness disparity measure into a single supervisor signal, using a proportional penalty above an empirically motivated parity threshold. \mathrm{synth\_score}=\mathrm{quality}\times\mathrm{fairness\_mult} where \mathrm{fairness\_mult}=\begin{cases}1.0&\text{if }\max_{\text{group}}\,\mathrm{FPR}/\min_{\text{group}}\,\mathrm{FPR}\leq 2\\[4.0pt] \min\!\left(1,\,\tfrac{2}{\mathrm{max\_rel\_FPR}}\right)&\text{otherwise.}\end{cases} Here, \mathrm{quality} is the SingleTableQualityReport overall score from SDMetrics, averaging marginal column-shape fidelity and pairwise-correlation fidelity into a single [0,1] value. \mathrm{max\_rel\_FPR} is the ratio of the highest to lowest group-wise false positive rate, computed via Fairlearn’s MetricFrame under a train-on-synthetic, test-on-real (TSTR) paradigm. A logistic regression is trained exclusively on synthetic rows and tested on held-out real data, so the FPR reflects how faithfully the synthetic distribution transfers to downstream classification across demographic groups. A ratio at or below 2\times is treated as acceptable parity, and beyond that the multiplier discounts quality proportionally. This ensures that a synthesizer cannot score well on fidelity alone if the synthetic data encodes unequal downstream error across groups. Experimental setup. We evaluate our tool on an open-source schizophrenia tabular dataset accessed via OpenML [ 3 ] . The dataset documents intersectional disparities in mental health diagnosis experienced by non-majority groups [ 7 , 8 , 15 ] . The table includes demographic attributes ( Race , Sex ), clinical context fields, numeric scales, and a binary Diagnosis label (schizophrenia vs. affective disorder). We fit 1,000 real rows, sample 500 synthetic rows, and run each synthesizer for 20 epochs. The stored metric outputs are read from committed JSON reports and can be reproduced deterministically by the bundled batch evaluator. User goal Real data Supervisor Generator CTGAN | TVAE | GaussianCopula Evaluator (TSTR) SDMetrics | Fairlearn FPR Composite Fairness Utility Score Data prepared for export synthetic data test set refine Figure 1: Memisis separates synthesis from evaluation so scoring cannot influence generation. The supervisor routes the user’s goal to a generator subgraph (CTGAN, TVAE, or GaussianCopula via SDV) and a separate evaluator subgraph (SDMetrics quality + Fairlearn FPR by group). After evaluation the supervisor compares the composite synth_score against repository-derived thresholds and either reports results or issues a refinement instruction back to the generator. Clients data owners | researchers | reviewers Interface Streamlit UI (Streamlit Cloud / GitHub) Web server train | sample | evaluate | models Agents LangGraph / ReACT Tools SDV | SDMetrics | Fairlearn Storage models .pkl | synthetic .csv | JSON reports LLama 3.2 (Ollama) NVIDIA RTX 3090 (training) Figure 2: Memisis deployment stack. Users (researchers, data owners) interact via Streamlit. All interfaces

Read the complete text