Self-supervised pretraining for an iterative image size agnostic vision transformer

Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu, Luc Van Gool · 2026

Read Self-supervised pretraining for an iterative image size agnostic vision transformer by Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu, Luc Van Gool (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-20392.txt.

Introduction

Prisadnikov and three colleagues build a self taught pretraining scheme for an iterative vision Transformer for any image size. Standard Transformers scale poorly with image size, leaving DINO stuck at low detail. A foveal inspired model instead handles fixed size multi zoom patches in sequence, without backpropagation through time. Their sequential to global self taught frame, built on DINO self distillation, pretrains size agnostic encoders at scale. ImageNet 1K and downstream tasks match rivals while compute stays flat at any input size.

Catalog record

CREATORNedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu, Luc Van Gool
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47936
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.20392

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-supervised pretraining for an iterative image size agnostic vision transformer Nedyalko Prisadnikov, Danda Pani Paudel, Yuqian Fu, Luc Van Gool Abstract Vision Transformers (ViTs) dominate self-supervised learning (SSL). While they have proven highly effective for large-scale pretraining, they are computationally inefficient and scale poorly with image size. Consequently, foundational models like DINO are constrained to low-resolution processing. A recent foveal-inspired transformer [ 44 ] achieves resolution agnosticism by iteratively processing a fixed-size context of multi-zoom patches. This model demonstrated promising results via supervised learning, utilizing a sequential, recurrent-like process without backpropagation through time. To unlock its potential as a foundational backbone, we introduce a novel sequential-to-global SSL framework based on DINO’s self-distillation objective. Supported by an efficient integral-image patch extraction method, our approach enables large-scale pretraining for image-size agnostic vision encoders. We achieve competitive performance on ImageNet-1K and downstream classification tasks, maintaining a constant computational budget regardless of input resolution. Figure 1 : Computational Efficiency and Scaling. (Left & Middle) Latency and GFLOPs of a standard ViT versus our foveal model (evaluated sequentially over 8 steps with a learned gaze policy). While standard ViT compute scales quadratically with the number of patches, our dynamic model maintains a strictly \mathcal{O}(1) footprint. (Right) ImageNet-1K Top-1 accuracy across extreme evaluation resolutions. All models are pretrained using an effective global crop size of 256\times 256 . While standard ViTs collapse at ultra-high resolutions ( > 1024) due to patch distribution shift, our image-size agnostic approach maintains robust performance, outperforming the supervised baseline [ 44 ] by +10\% . 1 Introduction Self-supervised learning (SSL) serves as the backbone of modern foundational models, driving breakthroughs in both computer vision [ 9 , 25 , 10 , 26 , 41 , 51 ] and natural language processing (NLP) [ 17 , 5 , 34 , 43 , 46 , 32 , 47 ] . By alleviating the burden of large-scale manual annotation, SSL enables pretraining at an unprecedented scale. Within computer vision, this progress has been largely fueled through the Vision Transformer (ViT [ 19 ] ) architecture. ViT-based encoders have become the standard for state-of-the-art vision models [ 9 , 41 , 51 ] and the visual components of Multimodal Large Language Models (MLLMs) [ 45 , 62 , 54 , 20 ] . Despite this success, the standard ViT architecture presents significant challenges regarding efficiency. Modern visual streams – from high-resolution cameras to robotic sensors – contain tens or hundreds of megapixels. Yet, foundational models like DINO are largely restricted to pretraining on small resolutions (typically 224\times 224 ). This limitation stems from the architecture’s computational complexity, which scales poorly with the input image size. To address this, recent work [ 44 ] argues that vision encoders should be image-size agnostic and task-driven. It proposed a dynamic, active vision transformer inspired by the foveal structure of the human visual system. Instead of processing a full grid of patches at once, the model operates iteratively. At each step, it processes a small, fixed number of “multi-zoom” patches around a chosen gaze location, maintaining a set of learnable state tokens that act as an evolving internal memory between steps. Crucially, these multi-zoom patches are extracted in a top-down manner: their dimensions are defined as a ratio of the overall image size rather than a fixed absolute pixel count. Because standard ViTs extract fixed-size patches (e.g. 16\times 16 pixels), increasing input resolution alters the semantic content within a single patch, leading to a distribution shift [ 52 ] and performance collapse (Fig. 1 , right). By utilizing ratio-based patches, the foveal approach eliminates this distribution shift. A reinforcement learning policy then dictates the sequence of gaze locations, creating a recurrent-like system that accumulates scene understanding without backpropagation through time (BPTT) [ 58 ] . While this architecture is promising, its initial validation was limited to supervised learning on ImageNet-1k [ 16 ] . For such a dynamic foveal architecture to serve as a true foundational backbone, it must be compatible with self-supervised pretraining. However, adapting state-of-the-art SSL frameworks like DINO [ 9 ] to a recurrent-like, stop-gradient architecture is highly non-trivial due to the sequential nature of the model. Furthermore, the patch extraction mechanism in [ 44 ] scales computationally with image resolution. This creates a bottleneck that prevents true image-size agnosticism, particularly under the high-throughput demands of training. In this paper, we resolve both of these critical limitations. We demonstrate, for the first time, that a dynamic foveal transformer can be pretrained in a self-supervised manner without BPTT. We achieve this by radically adapting the DINO self-distillation objective. Standard DINO utilizes a spatial multi-crop setting [ 8 ] , matching local spatial crops to global spatial views. We formulate a novel sequential-to-global distillation process. We demonstrate that a sequence of foveal glimpses can be treated analogously to the standard random resized local crops of DINO, training the sequentially evolving state tokens to match the embeddings of a standard, global ViT teacher. This hybrid formulation teaches the model to dynamically assemble a global understanding of the scene from iterative, local multi-zoom views. The main contributions of this work are as follows: • We propose a sequential-to-global self-supervised pretraining framework for dynamic vision transformers. By mapping sequences of local foveal glimpses to global spatial views without BPTT, we achieve a model capable of evolving its scene representation iteratively while leveraging DINO’s self-distillation objective. • We extend the multi-zoom patch mechanism from [ 44 ] by introducing a dense foveal grid (Sec. 3.2 ) around the center of gaze. This provides a richer, high-resolution context for the self-attention mechanism. • We introduce an efficient patch extraction method (Sec. 3.2 ) based on integral images. By isolating the resolution-dependent computation to a single, one-time preprocessing step, the subsequent iterative extraction of foveal patches becomes constant with respect to image size. See comparison with [ 44 ] in Fig. 2 . We validate our learned representations on ImageNet-1K [ 16 ] through linear probing. Operating as a standard ViT, our ViT-Small equivalent achieves 76% top-1 accuracy, competitive with DINO [ 9 ] . Crucially, in its dynamic 8-step foveal mode, a learned gaze policy yields 75% accuracy – significantly outperforming random gazes (71.7%). Although standard classification limits the spatial search advantages of an active policy, these results prove our model can build accurate global representations from sparse, localized glimpses. Most importantly, it achieves this with a fixed per-step computational budget decoupled from input resolution. Consequently, our image-size agnostic approach avoids the standard ViT performance collapse when evaluated on significantly larger images compared to training (Fig. 1 ). 2 Preliminaries As outlined in the introduction, our method builds upon the dynamic, fovea-inspired vision transformer proposed by Prisadnikov et al. [ 44 ] . To keep this paper self-contained, this section briefly reviews the core architectural setting introduced in that work. Subsequently, in Sec. 3 , we present our contributions and detail the proposed self-supervised pretraining framework. 2.1 Overall Architecture The dynamic foveal encoder from [ 44 ] consists of three main components: 1) a Transformer backbone that serves as the primary visual processor; 2) a Task Head that decodes the transformer’s latent state into task-specific predictions (e.g., classification logits); and 3) a Gaze

Read the complete text