Diverse Image Priors for Black-box Data-free Knowledge Distillation

Tri-Nhan Vo, Dang Nguyen, Trung Le, Kien Do, Sunil Gupta · 2026

Read Diverse Image Priors for Black-box Data-free Knowledge Distillation by Tri-Nhan Vo, Dang Nguyen, Trung Le, Kien Do, Sunil Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-25794.txt.

Introduction

Vo and four colleagues propose DIP-KD for distilling a large teacher into a small student with no training data and only black-box access. Privacy and proprietary limits often hide the teacher's internals and data. DIP-KD synthesizes diverse image priors, sharpens sample distinctions with contrastive learning, and distills through a primer student using soft probabilities. Only top-1 predictions are needed from the teacher. Tests across 12 benchmarks reach state-of-the-art results, with ablations confirming diversity drives learning.

Catalog record

CREATORTri-Nhan Vo, Dang Nguyen, Trung Le, Kien Do, Sunil Gupta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34326
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.25794

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diverse Image Priors for Black-box Data-free Knowledge Distillation Tri-Nhan Vo, Dang Nguyen, Trung Le, Kien Do, Sunil Gupta Abstract Knowledge distillation (KD) represents a vital mechanism to transfer expertise from complex teacher networks to efficient student models. However, in decentralized or secure AI ecosystems, privacy regulations and proprietary interests often restrict access to the teacher’s interface and original datasets. These constraints define a challenging black-box data-free KD scenario where only top-1 predictions and no training data are available. While recent approaches utilize synthetic data, they still face limitations in data diversity and distillation signals. We propose Diverse Image Priors Knowledge Distillation ( DIP-KD ), a framework that addresses these challenges through a three-phase collaborative pipeline: (1) Synthesis of image priors to capture diverse visual patterns and semantics; (2) Contrast to enhance the collective distinction between synthetic samples via contrastive learning; and (3) Distillation via a novel primer student that enables soft-probability KD. Our evaluation across 12 benchmarks shows that DIP-KD achieves state-of-the-art performance, with ablations confirming data diversity as critical for knowledge acquisition in restricted AI environments. Keywords: Knowledge Distillation ⋅ \cdot Black-box ⋅ \cdot Data-free ⋅ \cdot Diversity. 1 Introduction The evolution of Artificial Intelligence has resulted in high-capacity models that serve as centralized repositories of specialized expertise. However, the substantial size of these models often limits their mobility and deployment in decentralized environments, such as mobile or edge devices. To bridge this gap, research on model compression seeks to transfer these expert insights into lightweight, portable agents. Knowledge distillation (KD) [ 11 ] is the primary framework for this pedagogical exchange, where an expert network (teacher) guides the learning of a novice network (student). In practical AI ecosystems, this information flow is frequently obstructed by systemic barriers that prevent transparent knowledge exchange. Stringent privacy regulations, such as HIPAA or GDPR [ 5 , 23 ] , prohibit access to original training data, particularly in sensitive domains, such as personal, health, and proprietary data. Furthermore, intellectual property or security concerns [ 6 ] often render the teacher as black-box , withholding internal signals like activations, gradients, or class probabilities, and returning only the top-1 predictions. These constraints define the black-box data-free KD (BBDFKD) problem: training a student to emulate an expert’s intelligence through a narrow interface without any historical data. Recent methods in BBDFKD [ 31 , 37 , 34 ] leverage synthetic data, but they still encounter two primary bottlenecks. First, they suffer from a lack of domain coverage: synthetic samples fail to replicate the hierarchical structures and semantic variation of natural environments, resulting in a distribution mismatch with the teacher’s expertise. Second, the distillation signals are suboptimal. By relying exclusively on hard top-1 indices, students are denied the dark knowledge of inter-class relationships—a cornerstone of KD essential for emulating expert logic. Within a collective intelligence framework, this resembles an agent attempting to reconstruct a diverse collective memory from discrete feedback, deprived of the historical data that shaped the expert’s knowledge. Our method. We propose Diverse Image Priors Knowledge Distillation ( DIP-KD ) to tackle the BBDFKD problem, i.e., training a student from a black-box teacher with only top-1 signal and without access to any real data. To improve KD, we incorporate diversity , i.e., how broadly the samples distribute, into a collaborative three-phase pipeline: (1) Synthesis : crafts synthetic image priors with diverse patterns and semantics of natural objects; (2) Contrast : employs a primer student to act as a white-box mediator and ensure synthetic samples are informationally distinct via contrastive learning; and (3) Distillation : distills the final student optimally with hard labels from teacher and soft probabilistic labels from the primer student. Contribution. In summary, we highlight our contributions as follows: 1. We introduce image priors , a novel class of synthetic images exhibiting diversity across hierarchical structures, nonlinearity, and semantics. 2. We propose the novel use of a primer student , enabling contrastive optimization and extraction of soft probabilistic signals originally restricted in black-box KD. 3. Extensive analysis of our framework demonstrates that it effectively expands the semantic coverage of synthetic domains, leading to our state-of-the-art performance across 12 benchmarks. 2 Related Works 2.1 Foundations of Knowledge Transfer The pioneering idea in knowledge transfer between neural networks, intuitively, was to train a student to mimic the teacher’s outputs [ 1 ] . This concept evolved into knowledge distillation [ 11 ] , which proposed using soft probabilistic outputs, which are more informative than hard class labels. These soft labels act as essential inter-class hints, conveying the hidden relationships that define an expert’s logic. While foundational research has expanded these mechanisms, they generally assume a transparent environment where agents have full access to both original training data and the teacher model [ 24 , 22 ] . 2.2 Constraints in Data Availability When access to the primary data source is prohibited, researchers have explored data-efficient approaches. Few-shot KD methods such as [ 30 , 21 , 29 ] significantly reduce the required data volume but still rely on a small portion of the original source. Alternatively, other methods use proxy datasets to replace the source dataset [ 2 , 26 ] . However, they often have a strong built-in bias that could be mismatched with the teacher’s domain, as noted in [ 27 ] . Meanwhile, true data-free KD approaches often rely on dissecting the teacher’s internal structure or leveraging its gradients as guidance to harvest knowledge [ 3 , 8 , 7 , 28 ] . 2.3 Navigating Restricted Expert Interfaces Still, data-free methods collapse when the expert operates as a black-box, returning only hard labels as a top-1 prediction. The earliest method to address BBDFKD is ZSDB3 [ 31 ] , which assumes class robustness and image fidelity based on the distance to the decision boundary. This method uses random pixel-wise uniform noise for initialization and gradient estimation. However, high-dimensional boundaries can be over-complex for distance-based approaches, so obtaining diverse image samples scales exponentially worse with high-resolution images or numerous classes, limiting their success to small-scale, simple datasets. Subsequently, IDEAL [ 37 ] and DFHL-RS [ 34 ] utilize an extra generator network for more flexible data generation. Both assume that as the student improves during KD, its gradients could guide generator training. The methods employ in-class similarity and class-balance objectives for the generator, which is standard in data-free KD [ 3 ] . However, relying on the signals of an immature student is not reliable enough to train the generator. This setup has been observed to cause “overfitting of synthetic data” to the student, resulting in possible mode collapse and insufficient diversity [ 34 ] . Moreover, they are restricted to using hard labels instead of probabilities for KD, limiting the distillation signals. 3 Framework Figure 1 : Illustration of our method DIP-KD of three phases. ( 1 ) Synthesis : We craft image priors \mathcal{X}_{\mathcal{P}} with three sub-components: hierarchical noise, nonlinear transformation, and semantic cutmixing. Synthetic images x\sim\mathcal{X}_{\mathcal{P}} have diverse and distinct patterns. ( 2 ) Contrast : We construct the dataset \mathcal{D}=\{x|x\sim\mathcal{X}_{\mathcal{P}}\} and train a primer student S_{0} as a feature extractor. Then, we enforce an instance-discriminator R to distinguish embeddings of images x\sim\mathcal{D} to be similar to its positive

Read the complete text