Distilling Vision Transformers for Distortion-Robust Representation Learning
Konstantinos Alexis (✉), Giorgos Giannopoulos, Dimitrios Gunopulos · 2026
Read Distilling Vision Transformers for Distortion-Robust Representation Learning by Konstantinos Alexis (✉), Giorgos Giannopoulos, Dimitrios Gunopulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22529.txt.
Introduction
Alexis, Giannopoulos, and Gunopulos show how to learn visual codes that survive distortion when clean images are scarce or absent. Teacher and student start from the same pretrained vision model, but the teacher sees clean images while the student sees distorted versions. Their one-sided training aligns global codes, patch-level features, and attention maps across the pair. The student thus nears clean-image codes without ever viewing clean data directly. Across sorting tasks and varied damage, the student beats current rivals under the same amount of human supervision.
Catalog record
| CREATOR | Konstantinos Alexis (✉), Giorgos Giannopoulos, Dimitrios Gunopulos |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36154 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22529
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Distilling Vision Transformers for Distortion-Robust Representation Learning Konstantinos Alexis (✉), Giorgos Giannopoulos, Dimitrios Gunopulos Abstract Self-supervised learning has achieved remarkable success in learning visual representations from clean data, yet remains challenging when clean observations are sparse or not available at all. In this paper, we demonstrate that pretrained vision models can be leveraged to learn distortion-robust representations, which can then be effectively applied to downstream tasks operating on distorted observations. In particular, we propose an asymmetric knowledge distillation framework in which both teacher and student are initialized from the same pretrained Vision Transformer but receive different views of each image: the teacher processes clean images, while the student sees their distorted versions. We introduce multi-level distillation that aligns global embeddings, patch-level features, and attention maps and show that the student is able to approximate clean-image representations despite never directly accessing clean data. We evaluate our approach on image classification tasks across several datasets and under various distortions, consistently outperforming existing alternatives for the same amount of human supervision. 1 Introduction Learning robust visual representations from images under noise, occlusion, blur, or other distortions remains a central challenge in computer vision. In many real-world applications, such as medical imaging, remote sensing, and autonomous systems, high-quality, clean data is often scarce, expensive to collect, or altogether unavailable. As a result, models must often operate directly on corrupted observations, where extracting meaningful semantic features becomes substantially more difficult. Supervised learning methods, typically relying on clean and labeled datasets, tend to significantly degrade when trained or evaluated on such distorted inputs. Recent advances in self-supervised vision models, particularly Vision Transformers (ViT) [ 7 ] , have led to powerful encoders trained on massive image datasets that have demonstrated impressive performance across a wide range of image understanding tasks. However, when applied directly to corrupted inputs, these models struggle to preserve their semantic representations, as distortions disrupt both low-level visual cues and higher-level structural information. This raises the question: can we recover the clean representations of a pretrained encoder while only observing distorted versions of the images? In this work, we investigate this question and show that it is possible to approximate the clean feature space of a strong pretrained encoder, producing robust representations that are useful across diverse downstream tasks which lack access to clean data. Knowledge distillation has emerged as a standard paradigm for transferring knowledge from a teacher to a student model, often leading to improvements in model compactness, generalization, and training efficiency across various vision tasks including image classification, detection, and segmentation. Although recent studies have begun to explore its benefits in improving model robustness, its potential for learning effective representations under severe visual distortions remains underexplored. A notable exception is the work of Ravula et al. [ 24 ] , which leverages a pretrained contrastive encoder to recover representations of corrupted images, demonstrating that pretrained models can serve as effective priors for representation recovery under severe distortions. In this paper, we provide a simple yet effective knowledge distillation approach for training robust image encoders capable of handling highly distorted images. In particular, we propose a ViT-based knowledge distillation framework where a pretrained teacher processes clean images while the student model observes only their distorted counterparts, addressing an asymmetric learning setting where both networks are initialized from the same pretrained Vision Transformer but observe fundamentally different views of each image. Extending beyond contrastive representation recovery [ 24 ] , we introduce a multi-level distillation objective that aligns global image embeddings, patch-level features, and attention maps, providing richer and more structured supervision that guides the student to learn distortion-invariant representations without ever observing clean data. We assess the robustness and transferability of the learned image encoders through classification and transfer learning experiments across various datasets and distortion types, including random missing pixels, as well as Gaussian noise and blur. We show that given the same amount of labeled data, our method consistently outperforms supervised baselines and state-of-the-art self-supervised approaches, demonstrating that robust and transferable representations can be effectively obtained from corrupted inputs. Altogether, our main contributions are as follows: • We propose a knowledge distillation framework that leverages pretrained ViT models to train robust encoders for highly distorted images. • We introduce and combine multi-level distillation objectives, namely, global embeddings, patch-level representations, and attention maps, that provide stronger supervision and improve alignment between teacher and student. • We demonstrate through extensive experiments the robustness and generalization of learned encoders, achieving significantly improved results over existing baselines. 2 Related Work 2.1 Robust Image Recognition Despite impressive performance on clean images, deep neural networks have been shown to be fragile to simple corruptions. Early work revealed the severe degradation of image classifiers when evaluated on common perturbations such as noise, blur, and compression artifacts [ 5 , 12 ] , even after fine-tuning on these specific corruptions [ 6 ] . [ 8 ] further demonstrated that corruption-specific training fails to generalize to unseen corruptions. To enable systematic evaluation, [ 12 ] introduced ImageNet-C, which has since become a standard benchmark for robustness to distributional shifts, spanning diverse corruption types and severity levels. A prominent line of work addresses distortion robustness through data augmentation, ranging from simple techniques like random cropping and flipping to more sophisticated strategies. Methods such as Mixup [ 33 ] and AugMix [ 15 ] improve generalization by interpolating between training examples or composing diverse augmentation chains, exposing models to a wider range of corruptions at training time. Complementary approaches include feature normalization schemes and architectural design choices that promote invariance to input variations [ 27 , 20 ] . More recently, it has been observed that finetuning models pretrained on large-scale datasets, rather than training end-to-end from scratch, can substantially improve robustness to distribution shift [ 13 , 30 ] , indicating that rich pretrained representations carry inherent robustness benefits. Nevertheless, simply finetuning on corrupted images remains insufficient in highly degraded regimes, motivating more principled approaches for representation learning under distortion. 2.2 Denoising and Image Restoration Methods A parallel body of work approaches corrupted inputs through explicit image reconstruction, aiming to recover a clean image from its degraded observation. Classical denoising methods [ 1 ] have been succeeded by deep learning approaches such as FFDNet [ 34 ] , with more recent methods leveraging transformers and diffusion models for high-quality restoration [ 32 , 17 ] . Self-supervised denoising methods such as Noise2Noise [ 19 ] and Blind-Spot Networks [ 18 ] have further relaxed the requirement for clean training targets. However, pixel-level reconstruction is an indirect objective for downstream recognition. In contrast, we bypass pixel reconstruction entirely and operate directly in the latent space of a pretrained encoder, recovering clean semantic representations from corrupted inputs with the explicit goal of learning distortion-robust encoders for downstream tasks. 2.3 Self-Supervised Representation Learning Self-supervised learning (SSL) has produced a new generation of visual encoders that learn rich semantic representations without human annotation. Contrastive methods such as SimCLR [ 3 ] learn representations by attracting views of the same image while repelling views of different images. Non-contrastive approaches including BYOL [ 9 ] and SimSiam [ 4 ] achieve similar results without negative pairs. Masked image modeling methods such as MAE [