PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors
Brian Moser, Shalini Sarode, Federico Raue, Stanislav Frolov, Krzysztof Adamkiewicz, Arundhati Shanbhag, Joachim Folz, Tobias Nauen, Andreas Dengel, [-3pt] German Research Center for Artificial Intelligence (DFKI) · 2025
Read PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors by Brian Moser, Shalini Sarode, Federico Raue, Stanislav Frolov, Krzysztof Adamkiewicz, Arundhati Shanbhag, Joachim Folz, Tobias Nauen, Andreas Dengel, [-3pt] German Research Center for Artificial Intelligence (DFKI) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09905.txt.
Introduction
Brian B. Moser and eight co-authors present PRISM, a dataset distillation framework meant to keep synthetic training data diverse. Distillation compresses a big dataset into a small synthetic one, but methods that copy a single teacher model inherit its biases. As distilled size grows, samples turn smooth and alike, hurting generalization to new models.
PRISM splits the supervision: one teacher architecture supplies the logit-matching target, while a changing subset of other architectures handles the batch-normalization alignment objective. On ImageNet-1K it consistently beat single-teacher and recent multi-teacher methods at low and middle images-per-class levels. Feature measurements confirmed richer diversity within each class, and the authors add teacher selection analysis plus a parallel batch scheme for faster synthesis.
Catalog record
| CREATOR | Brian Moser, Shalini Sarode, Federico Raue, Stanislav Frolov, Krzysztof Adamkiewicz, Arundhati Shanbhag, Joachim Folz, Tobias Nauen, Andreas Dengel, [-3pt] German Research Center for Artificial Intelligence (DFKI) |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-09905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PRISM: Diversifying Dataset Distillation by Decoupling Architectural Priors Brian Moser, Shalini Sarode, Federico Raue, Stanislav Frolov, Krzysztof Adamkiewicz, Arundhati Shanbhag, Joachim Folz, Tobias Nauen, Andreas Dengel, [-3pt] German Research Center for Artificial Intelligence (DFKI) Abstract Dataset distillation (DD) promises compact yet faithful synthetic data, but existing approaches often inherit the inductive bias of a single teacher model. As dataset size increases, this bias drives generation toward overly smooth, homogeneous samples, reducing intra-class diversity and limiting generalization. We present PRISM (PRIors from diverse Source Models), a framework that disentangles architectural priors during synthesis. PRISM decouples the logit-matching and regularization objectives, supervising them with different teacher architectures: a primary model for logits and a stochastic subset for batch-normalization (BN) alignment. On ImageNet-1K, PRISM consistently and reproducibly outperforms single-teacher methods (e.g., SRe2L) and recent multi-teacher variants (e.g., G-VBSM) at low- and mid-IPC regimes. The generated data also show significantly richer intra-class diversity, as reflected by a notable drop in cosine similarity between features. We further analyze teacher selection strategies (pre- vs. intra-distillation) and introduce a scalable cross-class batch formation scheme for fast parallel synthesis. 1 Introduction Dataset distillation (DD) has emerged as a critical and controllable data generation method in modern deep learning, motivated by three core objectives: enhanced robustness against adversarial attacks ( 10 ) , improved privacy through membership and model inversion safeguards ( 7 ; 1 ) , and efficient data compression ( 27 ; 26 ) . Unlike large generative visual models such as diffusion models, DD produces images whose class semantics are guaranteed by gradient supervision, i.e. , they produce training-signal-equivalent samples ( 6 ; 9 ; 2 ) . As DD techniques have matured, a significant challenge remains unresolved: the failure to synthesize diverse, large-scale datasets that capture the complexity of their real-world counterparts. Current DD methods, including gradient matching ( 25 ) or parameter matching ( 2 ; 5 ) , also in combination with generative priors ( 3 ; 13 ; 19 ; 8 ) , predominantly concentrate on high-resolution synthetic images in small-scale settings, such as 50 or 100 images-per-class (IPC) and fail to close the gap between the full and compressed datasets. In this work, we question the motivation of compressing a dataset, since memory is cheap , in favor of the other motivations, namely robustness and privacy. Yet, naively upscaling DD techniques for an identical dataset size often results in overly smooth, feature-restricted datasets lacking sufficient diversity ( 15 ; 20 ; 17 ) , as illustrated in Figure 2 . This leads to homogeneous representations devoid of sufficient intra-class variability , thus impairing robustness as well as safety and ultimately limiting the practical utility of DD ( 18 ) . Our central argument is that knowledge within a trained network is inseparable from the architecture that contains it ( 22 ; 15 ) . Any single model possesses a strong inductive bias, i.e. , its “view” of the world. Distilling a dataset through one model inevitably imprints this single, limited view onto the synthetic data, resulting in a homogenous dataset that fails to train generalizing models. In order to create a truly generalizable synthetic dataset, we must synthesize it from a distribution of “world views”. Addressing this limitation, we propose diversifying the distillation process through a multi-architectural prior framework that simultaneously optimizes logit matching ( 25 ) and regularization through batch normalization (BN) alignment ( 23 ) with at least two decoupled teacher models, which we coin PRISM ( PRI ors from diverse S ource M odels). As such, our teacher decoupling is orthogonal to all existing scaling attempts, as shown in Figure 2 . Instead of relying on training schedules ( 17 ) , data initialization ( 20 ) , or post-evaluation pipelines ( 16 ) , PRISM introduces an orthogonal mechanism by decoupling the logit-matching objective from the BN-alignment regularization . This allows different architectural priors - from multiple, distinct teacher models - to simultaneously contribute to different aspects of the synthesis, complementing rather than replacing prior innovations. In summary, our contributions are clear and well-scoped within the ongoing evolution of dataset distillation: • First, we introduce PRISM, a novel framework that tackles the lack of diversity in dataset distillation by decoupling the architectural priors used for logit supervision and BN-alignment regularization. • Second, we provide a systematic analysis of teacher-selection strategies, demonstrating that a pre-distillation selection of diverse teachers is highly effective. • Third, we show that PRISM not only sets new state-of-the-art results, achieving up to 70.4% top-1 accuracy with a ResNet-101 at IPC=100, but also generates datasets with quantifiably greater intra-class diversity, directly addressing the critical challenge of homogeneity in modern dataset distillation, while still maintaining a simple and massively parallelizable synthesis pipeline that scales efficiently to large datasets like ImageNet-1K. Figure 1 : UMAP visualization of synthetic images from ImageNet-1K (10 classes), comparing SRe2L with our proposed multi-teacher alignment. Our approach, PRISM, generates significantly greater intra-class diversity, contrasting the overly uniform clusters of SRe2L that can lead to model overfitting more easily. Figure 2 : The core idea behind PRISM ( PRI ors from diverse S ource M odels): Use multiple, diverse models for decoupling the logit maximization and regularization through BN alignment instead of one, like in SRe2L and related work. 2 Related Work Dataset Distillation. The field has largely bifurcated into two main strategies. Early and prominent methods relied on batch-to-batch matching , which involves expensive bi-level optimization to align gradients ( 27 ) , training trajectories ( 2 ; 5 ) , or feature distributions ( 26 ) between real and synthetic data batches. While effective, the computational overhead of these methods makes them challenging to apply to large-scale datasets. Thus, the community has shifted towards batch-to-global matching , a paradigm pioneered by SRe2L ( 25 ) . This approach uses a pre-trained teacher model to generate global statistics ( e.g. , from batch normalization layers) and then optimizes synthetic images to match these global targets. This strategy is significantly more efficient and has enabled distillation at the scale of ImageNet-1K. However, because every synthetic image in a class is optimized against the same global signal, this approach often suffers from a lack of intra-class diversity ( 15 ) . Strategies for Enhancing Diversity. Recognizing the diversity challenge in batch-to-global matching , several methods have proposed alternative solutions. G-VBSM ( 15 ) was a key step in this direction, introducing the idea of using multiple teacher models and statistical metrics (beyond just BN statistics) to create a richer, more varied supervision signal. EDC ( 16 ) further refined this multi-teacher approach by identifying a suite of critical, and often overlooked, design choices in the post-training and evaluation pipeline, such as smoothing learning rate schedules and EMA-based evaluation, that are necessary to unlock the full potential of a diverse distilled dataset. Another related direction is D3S ( 12 ) , which employs an ensemble of independently trained ResNet-18 teachers and rotates them during optimization to reduce single-model bias. While similar in spirit, it keeps a fixed architecture family and couples the same ensemble for both soft-labeling and feature-matching, whereas PRISM decouples both the teacher roles and the architectural priors , enabling a richer