Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation – Extended Version

Muhammad Rajabinasab, Anton D. Lautrup, Arthur Zimek, University of Southern Denmark, Odense, Denmark · 2025

Read Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation – Extended Version by Muhammad Rajabinasab, Anton D. Lautrup, Arthur Zimek, University of Southern Denmark, Odense, Denmark (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-09591.txt.

Introduction

Muhammad Rajabinasab, Anton Lautrup, and Arthur Zimek proposed new measures of how alike two data sets are in 2025. Judging data set similarity matters for machine learning and data mining. Past methods can be costly, narrow, or fussy about settings, and they rarely view a data set as a whole.

The authors give two new metrics and lay out the math behind them. Two uses show their value. One checks synthetic data against the real data it mimics. The other scores feature selection methods by how well the kept features preserve the whole. Theory and tests together, they argue, show the metrics work well for these jobs.

Catalog record

CREATORMuhammad Rajabinasab, Anton D. Lautrup, Arthur Zimek, University of Southern Denmark, Odense, Denmark
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-09591

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Metrics for Inter-Dataset Similarity with Example Applications in Synthetic Data and Feature Selection Evaluation – Extended Version Muhammad Rajabinasab, Anton D. Lautrup, Arthur Zimek, University of Southern Denmark, Odense, Denmark Abstract Measuring inter-dataset similarity is an important task in machine learning and data mining with various use cases and applications. Existing methods for measuring inter-dataset similarity are computationally expensive, limited, or sensitive to different entities and non-trivial choices for parameters. They also lack a holistic perspective on the entire dataset. In this paper, we propose two novel metrics for measuring inter-dataset similarity. We discuss the mathematical foundation and the theoretical basis of our proposed metrics. We demonstrate the effectiveness of the proposed metrics by investigating two applications in the evaluation of synthetic data and in the evaluation of feature selection methods. The theoretical and empirical studies conducted in this paper illustrate the effectiveness of the proposed metrics. 1 Introduction Measuring similarity is an important task in machine learning and data mining with a large number of applications and use cases associated with it. It helps identify patterns, relationships, and structures within data. We often measure the similarity between data points in a dataset for applications such as pattern recognition [ 13 ] , outlier detection [ 45 ] , information retrieval [ 9 ] and recommendation systems [ 36 ] . Similarity can also be measured between features instead of data points to carry out the feature selection task [ 22 ] . A key research procedure in scientific practice is to measure the similarity between datasets. Consider comparing patient or consumer groups, identifying if experimental results are consistent with each other and with the current paradigm, or checking if a perturbation to an existing dataset caused a significant loss of information. One of the most simple applications of inter-dataset similarity in machine learning and data mining is data augmentation [ 41 ] , where similar datasets are identified and used to enhance the training set [ 11 ] . Other applications of inter-dataset similarity include synthetic data utility evaluation [ 7 , 3 ] , subspace quality assessment [ 33 , 18 ] , transfer learning [ 37 ] , federated learning [ 4 ] , instance space analysis [ 34 ] , and dataset selection [ 21 ] . In this paper, we propose two novel metrics for inter-dataset similarity based on Principal Component Analysis (PCA). PCA [ 26 ] is a well-established dimensionality reduction technique which is used widely in different fields of machine learning and data mining [ 17 , 42 , 14 , 10 , 40 , 31 ] . Our metrics, namely difference in explained variance and angle difference , can be used to measure different aspects of inter-dataset similarity. They can also be utilized to carry out different tasks. The primary contribution of this paper is the proposed metrics together with an in-depth exploration and discussion of two example applications for the new metrics, namely the evaluation of utility in synthetic data and the evaluation of performance of feature selection algorithms. We also explore their effectiveness in the sanity check of the privacy of synthetic data. Implementation and results are available on GitHub . Even though the applications we present in the paper seem to be specialized, we expect that the functionality of the proposed metrics extends far beyond these use cases to other areas of machine learning and data mining. The rest of the paper is structured as follows: In Section 2 , we briefly review previous work in inter-dataset similarity. In Section 3 , we discuss the theoretical background of PCA. In Section 4 , we introduce and analyze the proposed metrics. In Sections 5 and 6 , we explore and discuss two example applications of the proposed metrics, namely the evaluation of synthetic data and the evaluation of feature selection algorithms in detail. Finally, in Section 7 , we discuss limitations of the proposed metric and perspectives for future work. 2 Related Work Measuring inter-dataset similarity has always been an important task in the field of machine learning and data mining. It has been used both directly [ 37 ] and embodied in other metrics and measures [ 4 ] . The recent interest in AI-generated content such as images and text has mandated tools for measuring the authenticity hereof. One example of such a measure is the Fréchet Inception Distance [ 8 ] , which compares the distribution of one set of images (fakes) to the distribution of another (ground truth). Other metrics such as Kernel Maximum Mean Discrepancy [ 6 ] and Wasserstein distance [ 25 ] have also been used to measure a notion of similarity between two datasets. Measures such as Sammon Stress [ 33 ] and Kruskal Stress [ 18 ] are used for evaluating the goodness of fit in the low-dimensional subspaces by assessing distance- and topology preservation respectively. In transfer learning, inter-dataset similarity is used to ensure that the source and target datasets are similar [ 37 ] . This makes the transfer of knowledge more effective and improves the performance of downstream tasks. Inter-dataset similarity is also used in federated learning to accurately measure the similarity of the datasets from different silos [ 4 ] . This helps in improving the aggregation process and leads to a better performance for the global model. One other application of inter-dataset similarity is to find the most different dataset for the empirical analysis of an arbitrary machine learning or data mining method [ 21 ] to ensure a robust and comprehensive empirical analysis. Existing metrics can measure a notion of similarity between two datasets, but they have their own challenges and shortcomings. Fréchet Inception Distance is model-dependent as it relies on a pre-trained inception model. Maximum Mean Discrepancy is computationally expensive and depends on the choice of the kernel which can be non-trivial. Wasserstein distance has a high computational complexity and is sensitive to outliers. Sammon and Kruskal Stress also have a high computational complexity, Sammon Stress is sensitive to initialization and has scalability issues. Kruskall’s stress can be hard to interpret in terms of the quality of the dimensionality reduction techniques. It is also designed for Non-Metric Multi-Dimensional Scaling which focuses on preserving the rank order of distances rather than the exact distances. On the other hand, similarity metrics other than distance metrics for tabular data, are limited to feature-wise or pair-wise metrics and few capture the overall structure of the datasets [ 3 , 7 ] . 3 Background PCA is based on the idea of linearly transforming data into a new orthonormal basis where the principal directions capture the variation of the data. Each principal component is a linear combination of contributions from each original variable, and transforming data in accordance is interpreted as a projection along the principal component. In noisy data, PCA has the effect of concentrating the signal into the first few principal components, which can be useful for dimensionality reduction and data analysis [ 39 , 12 ] by transforming data into a space where the elements are mutually uncorrelated. Even though PCA is a well-known and established method, we discuss its mathematical foundation and nomenclature as a

Read the complete text