PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models

Tadeusz Dziarmaga *, Marcin Kądziołka, Laura Musioł, Patryk Marszałek, Artur Kasymov, Piotr Kościelniak, Marcin Mazur *, † \dagger · 2025

Read PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models by Tadeusz Dziarmaga *, Marcin Kądziołka, Laura Musioł, Patryk Marszałek, Artur Kasymov, Piotr Kościelniak, Marcin Mazur *, † \dagger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18462.txt.

Introduction

Tadeusz Dziarmaga and co-authors speed up the scoring of image and text generators in a paper from two thousand twenty-five. Judging such models means weighing how faithful, varied, and new their samples are. A recent metric does this well on paper but costs too much compute to run at scale.

Their framework, PALATE, rests on a quirk in the law of total expectation, a basic probability rule. Paired with a standard distance metric and a recent image feature tool, it matched or beat top methods in their tests. It also ran faster and scaled to large data sets. The authors highlight its strength at spotting memorized samples and checking how well models generalize.

Catalog record

CREATORTadeusz Dziarmaga *, Marcin Kądziołka, Laura Musioł, Patryk Marszałek, Artur Kasymov, Piotr Kościelniak, Marcin Mazur *, † \dagger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18462

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PALATE: Peculiar Application of the Law of Total Expectation to Enhance the Evaluation of Deep Generative Models Tadeusz Dziarmaga *, Marcin Kądziołka, Laura Musioł, Patryk Marszałek, Artur Kasymov, Piotr Kościelniak, Marcin Mazur *, † \dagger Abstract Deep generative models (DGMs) have caused a paradigm shift in the field of machine learning, yielding noteworthy advancements in domains such as image synthesis, natural language processing, and other related areas. However, a comprehensive evaluation of these models that effectively captures the interplay among fidelity, diversity, and novelty in generated samples remains a persistent challenge. A recently proposed approach, the Feature Likelihood Divergence (FLD), offers a theoretically grounded and practical tool in this regard but faces notable computational limitations. In this paper, we propose PALATE, a novel framework designed to enhance the evaluation of DGMs by addressing the efficiency constraints of FLD. Our approach is based on a peculiar application of the law of total expectation to random variables representing accessible data. When integrated with the MMD baseline metric and the recently introduced DINOv3 feature extractor, PALATE provides a comprehensive evaluation framework that matches or exceeds state-of-the-art methods while offering superior computational efficiency and scalability to large-scale datasets. Through a series of experiments, we demonstrate that the PALATE enhancement constitutes a practical and computationally efficient framework for holistic DGM assessment, particularly in detecting sample memorization and evaluating generalization capabilities. The source code is provided as supplementary material. \nonumnote * These authors contributed equally to this work. \dagger Corresponding author: [email protected] . 1 Introduction In recent years, deep generative models (DGMs) have garnered significant attention, drawing interest not only from the machine learning community but also from researchers in other scientific disciplines, practitioners, and the general public [ 32 ] . DGMs have matured to the point of enabling strong downstream results, including photorealistic imagery [ 33 ] , speech that emulates authentic discourse [ 10 ] , and text reminiscent of human writing [ 28 ] . Figure 1 : We propose PALATE, a memorization-aware enhancement to deep generative model assessment. Grounded in a peculiar application of the law of total expectation and leveraging DINOv3 representations, it yields M_{\text{PALATE}} , an MMD-based holistic evaluation metric that captures fidelity, diversity, and memorization in a single score. Regardless of architecture, DGMs are trained to generate synthetic data that closely resemble real data. Since learning objectives are model-specific, fair post-training assessment is typically architecture agnostic and therefore sample-based, comparing generated samples to real data. The most prominent examples are Inception Score (IS) [ 35 ] and Fréchet Inception Distance (FID) [ 14 ] , popular due to usability, reasonable agreement with human perceptual judgments, and sensitivity to diversity. In particular, FID often captures fidelity by measuring a statistical distance between feature distributions fitted to real and generated samples. Nevertheless, despite its de facto role as a gold standard, FID generally lacks the capacity to detect overfitting or sample memorization [ 18 ] . More recent metrics such as Kernel Inception Distance (KID) [ 5 ] or Precision and Recall (P&R) [ 34 ] address other weaknesses of FID, such as bias, normality assumptions, or the inability to separate fidelity from diversity, but they still do not resolve memorization sensitivity. Complementary memorization metrics, including the C_{T} Score [ 26 ] and Authenticity [ 1 ] , target overfitting directly by quantifying training data reuse. While important, this separation of fidelity and diversity evaluation from memorization evaluation complicates model selection, especially in large-scale benchmarking where computational efficiency and a unified, actionable signal are critical. A desirable evaluation metric should therefore be holistic, providing a single value that jointly validates generated samples in terms of fidelity, diversity, and novelty, where novelty is the counterpart of memorization. A recent and promising step is Feature Likelihood Divergence (FLD) [ 18 ] , designed to assess generalization while preserving benefits of feature-based evaluation. Nevertheless, FLD can become computationally challenging on complex, large-scale datasets, because reliable estimation may require substantial numbers of samples (see our ImageNet results in Section 5 ). To address this issue, we propose a novel enhancement to the evaluation of DGMs, which we refer to as PALATE. Our solution is grounded in a p eculiar a pplication of the la w of t otal e xpectation (hence the name). The objective is to upgrade a baseline metric that already captures fidelity and diversity, making it sensitive to novelty and memorization while preserving the simplicity of sample-based evaluation (see Figure 1 for an overview). Our experiments show that PALATE, when paired with an MMD-based approach, yields a single-value holistic evaluation metric that is aligned with state-of-the-art solutions while offering improved computational efficiency. To the best of our knowledge, our work is the first to incorporate the DINOv3 feature extractor [ 36 ] into DGM evaluation. In summary, our work contributes the following: 1. We propose PALATE, a novel general enhancement for evaluating DGMs that upgrades a baseline metric to ensure sensitivity to training sample memorization while preserving fidelity and diversity assessment. 2. We provide a theoretical justification based on a peculiar application of the law of total expectation to random variables defined over a Euclidean data space. 3. We conduct extensive experiments on real-world datasets, introducing DINOv3 embeddings to DGM evaluation and showing that our approach yields a computationally efficient single-value holistic metric that matches, and sometimes exceeds, state-of-the-art solutions. 2 Related Work In this section, we review state-of-the-art evaluation metrics for DGMs that are relevant to our work. Fidelity and Diversity Metrics Inception Score (IS) [ 35 ] is a sample-based metric that uses a pre-trained Inception-v3 network [ 38 ] and primarily reflects diversity, with limited ability to assess fidelity. Metrics targeting both fidelity and diversity include Fréchet Inception Distance (FID) [ 14 ] , Kernel Inception Distance (KID) [ 5 ] , and CLIP-MMD (CMMD) [ 17 ] . FID compares fitted feature distributions of real and generated samples, and it is sensitive both to biases induced by the embedding space and to the Gaussian approximation. In contrast, KID is MMD-based, avoids the Gaussian assumption, and is typically less biased. As part of a broader evaluation view, FD and KD [ 37 ] replace Inception features with DINOv2 embeddings, which can better capture global structure and salient objects, improving perceptual alignment. CMMD is also MMD-based, but uses CLIP embeddings and a Gaussian RBF kernel, differing from KID’s typical Inception-v3 embeddings and kernel choices. Improved Precision and Recall [ 21 ] and Density and Coverage [ 27 ] further decompose quality and coverage, providing more diagnostic feedback than a single scalar score. Beyond these, the Vendi Score [ 9 ] offers a flexible, similarity-based notion of diversity that can expose mode collapse and duplication effects even when coverage appears high. Other recent proposals, such as FKEA [ 30 ] and RKE Score [ 16 ] , provide alternative, often kernel or information theoretic, perspectives on fidelity and diversity. However, these metrics are generally ineffective at detecting memorization and overfitting. Memorization Metrics Concerns regarding DGMs memorizing training data have motivated metrics designed specifically to detect training data reuse. The C_{T} Score [ 26 ] quantifies the probability that a generated sample is indistinguishable from a real sample, while Authenticity [

Read the complete text