Distribution-aware Dataset Distillation for Efficient Image Restoration

Zhuroan Zheng, Xin Su, Chen Wu, Xiuyi Jia · 2025

Read Distribution-aware Dataset Distillation for Efficient Image Restoration by Zhuroan Zheng, Xin Su, Chen Wu, Xiuyi Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14826.txt.

Introduction

Zhuoran Zheng and co-authors present a 2025 method for training image repair models on far less data. Repair models for blur, noise, and weather damage need huge training sets, and dataset distillation, which shrinks a set to its essence, had not been tried for repair. Their method, TripleD, first grades images by complexity with a vision transformer, then picks a small subset.

A light network then nudges the subset so its feature spread matches the full set. Training runs in two stages: easy samples first to lay a base, harder ones later. The result trains a top repair model on 4K images with one consumer graphics card in under eight hours, a vast saving in compute.

Catalog record

CREATORZhuroan Zheng, Xin Su, Chen Wu, Xiuyi Jia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14826

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distribution-aware Dataset Distillation for Efficient Image Restoration Zhuroan Zheng, Xin Su, Chen Wu, Xiuyi Jia Abstract With the exponential increase in image data, training an image restoration model is laborious. Dataset distillation is a potential solution to this problem, yet current distillation techniques are a blank canvas in the field of image restoration. To fill this gap, we propose the Distribution-aware Dataset Distillation method (TripleD), a new framework that extends the principles of dataset distillation to image restoration. Specifically, TripleD uses a pre-trained vision Transformer to extract features from images for complexity evaluation, and the subset (the number of samples is much smaller than the original training set) is selected based on complexity. The selected subset is then fed through a lightweight CNN that fine-tunes the image distribution to align with the distribution of the original dataset at the feature level. Our method achieves promising performance on multiple image restoration tasks, including multi-task image restoration, all-in-one image restoration, and ultra-high-definition image restoration tasks. Note that we can train a state-of-the-art image restoration model on an ultra-high-definition (4K resolution) dataset using only one consumer-grade GPU in less than 8 hours ( 500 % savings in computing resources and immeasurable training time). 1 Introduction Figure 1 : Performance and analysis of our TripleD method for image restoration. First, the figure on the left shows that using only 2 % of the original dataset allows the model to perform close to training on the full set. The figure on the right demonstrates an ablation where the performance of the image restoration model rises steadily as the proportion of synthetic datasets rises, but the improvement is very limited. Image restoration provides high-quality information for downstream visual tasks such as denoising [ 7 ] , deblurring [ 14 ] , and deraining [ 41 ] . Recent advancements have employed deep learning techniques to boost restoration quality, achieving state-of-the-art results across various benchmarks [ 36 , 42 ] . Although deep learning algorithms have shown promising results, they are highly dependent on the quantity of the datasets. Indeed, it has become a trend to create large-scale benchmarks, which unfortunately also brings expensive training costs. To alleviate this problem, dataset distillation [ 32 ] becomes a potential option. This technique aims to synthesize a representative subset (the amount of data is much smaller than the original dataset) as a training set for the model to achieve similar performance when trained on a large-scale dataset. However, this technology is still a blank in the field of image restoration. Since the task of image restoration is a refined regression operation, there is no margin error like the classification task (for classification tasks, a probability of 0.5 to 1 indicates that the label exists). In this paper, we design a distribution-aware dataset distillation (TripleD) algorithm, which can efficiently synthesise a small dataset from a large-scale image restoration dataset. As illustrated in Figure 1 , TripleD mehrod, which has two properties: i) A high-quality synthetic dataset can be distilled for efficient training of the image recovery model using only 2% of the data volume of the training set (PSNR reaches the upper limit of around 95%). ii) Although the model performs better as the number of synthetic datasets increases, the improvement is limited. TripleD achieves up to 90 \sim 95% of the performance of full-dataset training while utilizing only 1 \sim 5% of the data. Additionally, TripleD significantly reduces the training resource requirements, enabling models such as Restormer to be trained efficiently on a single RTX 3090 GPU. Specifically, TripleD uses features extracted from downsampled images (the input images are all downsampled to a resolution of 128\times 128 ) using a Vision Transformer (ViT) [ 9 ] to represent the complexity of images. Here, this pre-trained Transformer is trained on the ImageNet dataset, and the label is the entropy value of the image. it is worth noting that relying only on the entropy value of the image to represent the image complexity is subject to error, and we manually conduct a significant amount of revision annotation. We use the complexity of images for key purposes: With the help of the GPU shader, Vision Transformer can be run faster on downsampled input images without losing the semantics of the original resolution image. Relying only on traditional calculations to obtain the entropy value of an image is not only time-consuming and laborious but also subject to errors. Indeed, we also considered measures such as cosine distance, standard deviation, KL divergence, and entropy, but they performed poorly in evaluating the complexity of images. For the generation of synthetic samples, we used a diffusion model, which produces better and more varied quality than GAN [ 8 ] . Finally, for distribution matching between the synthetic dataset and the selected dataset, we used a joint regularization i.e., KL scatter and L2. Our contributions are as follows: • To the best of our knowledge, we are the first to design a dataset distillation framework (TripleD) for image restoration tasks, which alleviates the cost of training image restoration models on large-scale data sets. • We develop an efficient dynamic distillation method that progressively increases the complexity of the sample bar to select high-quality sample clusters for distillation to obtain sub-datasets (2% of full dataset). • Our method can also achieve dataset distillation on a single GPU in a limited amount of time when executing ultra-high-definition datasets. Extensive experimental results show that our method can help the model maintain 90% of its performance at a lower cost. 2 Related Work Image Restoration. Image restoration aims to recover clean images from degraded inputs, addressing tasks such as denoising [ 40 ] , deblurring [ 19 ] , and deraining [ 38 ] . Recently, since Transformer has a strong ability to capture global features, models based on it have shown very promising results, such as Restormer [ 37 ] , SwinIR [ 16 ] and IPT [ 6 ] , have achieved state-of-the-art results. Unfortunately, these methods are too computationally expensive for deployment on mobile or embedded devices. So far, RAMiT [ 20 ] , and FreqFormer [ 23 ] , propose more efficient attention mechanisms to reduce complexity while maintaining performance. Although the inference efficiency of models has been significantly improved, the exponential increase in data sets has not been addressed in terms of training efficiency. Dataset distillation provides a potential solution to this problem. Subset Selection. Subset selection is a sampling technique for picking representative samples from a dataset, maintaining model performance with sampled data. Active learning [ 29 , 28 ] , continuous learning [ 22 , 18 ] , and incremental learning [ 26 ] all reduce training costs by selecting the samples with the greatest information content, such as the sample variance or entropy. For example, Settles [ 30 ] demonstrated how active learning can effectively prioritize data points, minimizing labeling efforts while maintaining high model performance. The Herding method in [ 3 ] iteratively minimizes the mean embedding discrepancy between the selected subset and the whole dataset in the feature space. Indeed, it is particularly important for image restoration tasks, which are usually high-cost due to its dense prediction. Recent advances in dynamic

Read the complete text