DEREC-SIMPRO : unlock Language Model benefits to advance Synthesis in Data Clean Room
Tung Sum Thomas Kwok, Chi-Hua Wang, Guang Cheng · 2024
Read DEREC-SIMPRO : unlock Language Model benefits to advance Synthesis in Data Clean Room by Tung Sum Thomas Kwok, Chi-Hua Wang, Guang Cheng (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00879.txt.
Description
Abstract Data collaboration via Data Clean Room offers value but raises privacy concerns, which can be addressed through synthetic data and multi-table synthesizers. Common multi-table synthesizers fail to perform when subjects occur repeatedly in both tables. This is an urgent yet unresolved problem, since having both tables with repeating subjects is common.
Catalog record
| CREATOR | Tung Sum Thomas Kwok, Chi-Hua Wang, Guang Cheng |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00879
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DEREC-SIMPRO : unlock Language Model benefits to advance Synthesis in Data Clean Room Tung Sum Thomas Kwok, Chi-Hua Wang, Guang Cheng Abstract Data collaboration via Data Clean Room offers value but raises privacy concerns, which can be addressed through synthetic data and multi-table synthesizers. Common multi-table synthesizers fail to perform when subjects occur repeatedly in both tables. This is an urgent yet unresolved problem, since having both tables with repeating subjects is common. To improve performance in this scenario, we present the DEREC 3-step pre-processing pipeline to generalize adaptability of multi-table synthesizers. We also introduce the SIMPRO 3-aspect evaluation metrics, which leverage conditional distribution and large-scale simultaneous hypothesis testing to provide comprehensive feedback on synthetic data fidelity at both column and table levels. Results show that using DEREC improves fidelity, and multi-table synthesizers outperform single-table counterparts in collaboration settings. Together, the DEREC-SIMPRO pipeline offers a robust solution for generalizing data collaboration, promoting a more efficient, data-driven society. Key Words: Data Collaboration, Data Clean Room, Large Language Models, Data Synthesis, Synthetic Data Evaluation Figure 1: Proposed DEREC 3-steps pre-processing pipeline and SIMPRO 3-aspects synthetic data evaluation metrics. (1) Party 1 and Party 2 contribute their data in Data Clean Room (2) Construct parent and child columns with DEREC (Section 4.1 ) (3) Generate synthetic data with language model-based multi-table synthesizer (4) Audit the quality of synthetic data through SIMPRO evaluation metrics (Section 4.2 ): (4.1) Compute statistical similarity between original and synthetic data (Section 4.2.1 ) (4.2 & 4.3) Compare synthetic data quality generated by different models (Section 4.2 ) using statistical equality test p-values (4.2) and distance metrics (4.3) (5) Augment the first (second) party real data with synthetic second (first) party data (6) Send the augmented tabular data back to both parties to share the data 1 Introduction Data collaboration promotes innovation by leveraging the collective expertise of various stakeholders, e.g., two initiatives in Nepal have developed a platform to gather resources from different parties to improve local health and population data systems [ 54 ] . Increasingly, enterprises are turning to Data Clean Rooms, which connect and synthesize datasets from multiple sources [ 37 ] [ 36 ] . Multi-table synthesizers address both data synthesis and integration needs, sparking interest in incorporating Language Model backbones for data generation, particularly following the success of ChatGPT [ 32 ] . In managing a generation-based Data Clean Room that involves synthesis and evaluation processes, a key challenge for each is identified. During the synthesis process, existing multi-table synthesizers require a strict one-to-many table relationship, and any deviation from this can result in underperformance. However, real-life datasets often feature subjects that occur repeatedly (Section 3.1 ), which existing synthesizers are not equipped to handle due to their inability to manage many-to-many relationships. This limitation negatively impacts synthesizer performance (Refer to Section 6.2 ). In the evaluation step, current metrics are inefficient in assessing synthetic data quality in multi-table contexts, e.g., the Synthetic Data Metrics (” SDMetrics ”) [ 11 ] tends to favor ’smoothed’ results without accurately reflecting shape similarity. The first challenge involves effectively establishing connections between two tables with many-to-many relationships, as the potential combinations for each column can be infinite unless additional information from the datasets helps differentiate specific observations. The second challenge focuses on defining an indicator that accurately measures the similarity between features of the original and synthetic datasets. In summary, the contributions of this work include: • Introducing a pipeline (Section 4.1 ) to transform many-to-many collaborative data into one-to-many scenarios (Figure 5 ) for data synthesis • Proposing the usage of conditional distribution (Section 4.2 ) and similarity metrics (Section 4.2.2 and 4.2.3 ) to measure individual column synthetic data fidelity • Developing an iterative algorithm to manage synthetic table fidelity in entirety (Section 4.2.1 ) using ’distribution of distribution similarity’ (Figure 12 ) 2 Related Works We refer to papers regarding data synthesis, where limited amount of work on multi-table cases leads to overlooking of certain details that significantly impact synthetic data quality. Privacy leakage when synthesising data. Research indicates that using transformers in data synthesis can enhance data security. While synthetic data aims to mitigate privacy leakage, studies show potential risks, particularly in ID-based data matching scenarios common in Data Clean Room [ 47 ] [ 18 ] . The SiloFuse framework [ 44 ] offers a privacy-protecting solution by using distributed transformers to convert data into latent representations, making it impossible to retrieve the original data without access to the transformer. Consequently, language model-based synthesizers, particularly those incorporating multi-table synthesizers with LLM backbones (Section 5.2 ), are explored as a potential solution for enhancing synthetic data security. Multi-table synthesizer architecture for data collaboration scenarios. Multi-table synthesizers are designed for tables with a one-to-many relationship, as illustrated as the parent-child structure in [ 38 ] [ 45 ] [ 44 ] . This architecture is often impractical in Data Clean Rooms, where formats like logbooks are frequently shared among users (Section 3.1 ). Despite significant advancements in single-table synthesizers like CT-GAN [ 58 ] and TabDDPM [ 23 ] , these models struggle in data collaboration contexts (Figure 9 ), highlighting the need for multi-table synthesizers (Table 1 ). Intuitive evaluation results for immediate performance monitoring. [ 11 ] and [ 44 ] assess synthetic data quality using robust mathematical formulas and provide score-based outputs. However, while plotting these results for comparison, we found that the metric scores are biased in certain scenarios (Section 3.2 ), highlighting the need for new metrics to address this bias. The Benjamini-Hochberg procedure [ 4 ] , commonly used in medical research, offers a generalized performance analysis across various factors. This inspired the development of the SIMPRO evaluation metrics, which provide a comprehensive analysis and utilize illustrative plots for immediate performance comparison. Table 1: SiloFuse provides privacy protection through tokenization but is limited to one-to-one data relationships. While REaLTabFormer supports one-to-many data synthesis with tokenization, the one-to-many assumption falls short in various collaborative data scenarios (Section 3.1 ). Tabular Data Singular Multiple Token- Generator Table Table -ization CT-GAN [ 58 ] ✓ \times \times TabDDPM [ 23 ] ✓ \times \times SiloFuse [ 44 ] ✓ \times ✓ REaLTabFormer [ 45 ] ✓ ✓ ✓ DEREC-REaLTabFormer ✓ ✓ ✓ (This work) 3 Issues of multi-table synthesizers in realistic data collaboration scenarios There are two concerns on multi-table synthesis with real life data, namely privacy and architecture, and one concern on multi-table synthesis evaluation, that is evaluation metrics intuitiveness. Section 2 shows that the privacy concern can be well addressed by using language model-based synthesizer [ 44 ] , such as the REaLTabFormer [ 45 ] . This work then focuses on the architectural issue of multi-table synthesizers and the intuitiveness of evaluation metrics. 3.1 Multi-table Synthesizer Architecture Cross-Table Management. Multi-table synthesizers are not designed to manage across tables without pre-defined hierarchy. In Data Clean Room, tables from two parties do not necessarily have a parent table that contains one observation for each unique identifier (Table 3 ). Formal definitions are in Appendix A . Figure 3 serves as an ideal parent table while Figure 3 acts as a child table. When both parties’ data contain repeating identifiers