Multi-Scale Visual Prompting for Lightweight Small-Image Classification

Salim Khazem · 2025

Read Multi-Scale Visual Prompting for Lightweight Small-Image Classification by Salim Khazem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-03663.txt.

Introduction

Salim Khazem adapts a prompting trick from giant vision models to the small images used in classrooms and prototypes. Visual prompting teaches frozen models new tasks through learnable patterns pasted onto inputs. His 2025 module, MSVP, learns prompt maps at three scales at once for tiny benchmarks like MNIST and CIFAR-10.

Global, mid-scale, and local prompts merge with the image through a single lightweight convolution. The addition costs a negligible sliver of extra parameters and works with plain CNNs, ResNet-18, and small Vision Transformers alike. Every backbone improved consistently across the three datasets. Ablations and visualizations show which scales carry the benefit at this resolution.

Catalog record

CREATORSalim Khazem
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-03663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Scale Visual Prompting for Lightweight Small-Image Classification Salim Khazem Abstract Visual prompting has recently emerged as an efficient strategy to adapt vision models using lightweight, learnable parameters injected into the input space. However, prior work mainly targets large Vision Transformers and high-resolution datasets such as ImageNet. In contrast, small-image benchmarks like MNIST, Fashion-MNIST, and CIFAR-10 remain widely used in education, prototyping, and research, yet have received little attention in the context of prompting. In this paper, we introduce Multi-Scale Visual Prompting (MSVP) , a simple and generic module that learns a set of global, mid-scale, and local prompt maps fused with the input image via a lightweight 1 × 1 1\times 1 convolution. MSVP is backbone-agnostic, adds less than 0.02 % 0.02\% parameters, and significantly improves performance across CNN and Vision Transformer backbones. We provide a unified benchmark on MNIST, Fashion-MNIST, and CIFAR-10 using a simple CNN, ResNet-18, and a small Vision Transformer. Our method yields consistent improvements with negligible computational overhead. We further include ablations on prompt scales, fusion strategies, and backbone architectures, along with qualitative analyzes using prompt visualizations and Grad-CAM. Our results demonstrate that multi-scale prompting provides an effective inductive bias even on low-resolution images. The code is available at https://github.com/salimkhazem/msvp.git I Introduction Small-image benchmarks such as MNIST [ 22 ] , Fashion-MNIST [ 32 ] and CIFAR-10 [ 20 ] have played a central role in the history of deep learning for computer vision. Despite their apparent simplicity, they continue to be used extensively for algorithmic prototyping, educational purposes, and controlled method comparisons. These datasets expose practical challenges related to generalization, calibration, and robustness, especially when models are constrained to be small and computationally efficient. At the same time, the last few years have witnessed a rapid shift towards parameter-efficient tuning methods such as LoRA [ 11 ] , adapters [ 10 ] , and prompt tuning [ 23 , 24 ] . Instead of fully fine-tuning all parameters of a large pre-trained model, many approaches propose to adapt only a small subset of additional parameters. In natural language processing, soft prompts and low-rank adapters have become standard tools. In computer vision, Visual Prompt Tuning (VPT) [ 12 ] and its extensions [ 33 , 1 , 13 , 17 ] show that Vision Transformers (ViTs) [ 6 ] can be adapted to new tasks by introducing learnable prompt tokens at the input or intermediate layers, while freezing the majority of the backbone. Most of this work, however, focuses on high-resolution tasks (e.g., ImageNet [ 19 ] ) and large pre-trained ViTs. In such setups, visual prompting serves primarily as a mechanism for task adaptation and transfer learning. In contrast, small-image datasets are often tackled with models trained from scratch , and the role of prompting in this regime is largely unexplored. This raises a natural question: Can we leverage the idea of visual prompting to obtain systematic gains on low-resolution benchmarks and small backbones, even when no large-scale pre-training is used? In this paper, we answer this question in the affirmative by proposing Multi-Scale Visual Prompting (MSVP) . Instead of introducing abstract prompt tokens in a sequence of patch embeddings, MSVP operates directly in pixel space . It maintains a small set of learnable prompt maps at different spatial scales (global, mid-scale, local), upsamples them to the image resolution, and fuses them with the original image via a 1\times 1 convolution. The result is a prompt-augmented image that can be fed into any backbone architecture. Fig. 1 : Multi-Scale Visual Prompting architecture. Three learnable prompts at different spatial scales (global 1\times 1 , mid 4\times 4 , local 8\times 8 ) are upsampled to input resolution and fused via element-wise addition. Our main hypothesis is that these multi-scale prompt maps act as a flexible, data-driven bias field that can guide the backbone towards more discriminative features, even on very small images. For instance, global prompts may encode class-dependent intensity or color shifts, mid-scale prompts may capture coarse shapes or object layouts, and local prompts may highlight detailed edges or textures. I-A Contributions The main contributions of this work can be summarized as follows: • We introduce MSVP , a simple and generic multi-scale visual prompting module that operates in pixel space and can be attached to any CNN or ViT backbone. The module is lightweight and adds fewer than 0.02\% parameters in our experiments. • We provide a unified empirical evaluation of MSVP on MNIST, Fashion-MNIST, and CIFAR-10, using three families of backbones (simple CNN [ 19 , 15 ] , ResNet-18 [ 8 ] , ViT-Tiny [ 31 , 30 ] ). This offers a coherent view of how prompting behaves in the small-image regime. • We conduct extensive ablation studies on (i) the choice of spatial scales, (ii) alternative fusion strategies, and (iii) backbone sensitivity, and we show that multi-scale prompting consistently outperforms single-scale variants. • We analyze the behavior of MSVP through qualitative visualizations , including learned prompt maps and Grad-CAM explanations [ 28 ] , illustrating how prompts influence attention and decision boundaries. Beyond the specific instantiation we propose, our results suggest that prompt-based inductive biases are not limited to large pre-trained models, but can also be useful in small, fully supervised setups. II Related Work Small-Image Benchmarks : MNIST [ 22 ] is one of the earliest and most widely used benchmarks for handwritten digit recognition, consisting of 28\times 28 grayscale images with 10 classes. Fashion-MNIST [ 32 ] was introduced as a more challenging drop-in replacement for MNIST, with the same image size and class count but more diverse visual patterns. CIFAR-10 [ 20 ] provides 32\times 32 RGB images from 10 object categories and remains a central benchmark for evaluating new architectures and regularization techniques. Despite their modest size, these datasets are still used in recent work for rapid experimentation and for didactic purposes, and they continue to reveal interesting behaviors in optimization and generalization.Recent work has also explored alternative representations for small images, such as polygonal approximations [ 15 ] , demonstrating that simplified representations can achieve competitive performance. In contrast, our approach maintains pixel-level representations but augments them with learnable multi-scale prompts. Backbone Architectures for Small Images : Convolutional networks, from early designs such as LeNet [ 21 ] to modern residual architectures like ResNet [ 9 ] , have dominated performance on small-image tasks. They naturally exploit local spatial correlations and translation equivariance. In parallel, Vision Transformers (ViTs) [ 5 ] have emerged as a strong alternative for image recognition by operating on sequences of patch embeddings. Although ViTs typically require strong regularization and/or pre-training to perform well, smaller variants with reduced patch size and depth have been successfully trained on CIFAR-scale datasets. In our experiments, we deliberately choose three representative backbones: a shallow CNN (for pedagogical interpretability), a standard ResNet-18 [ 8 ] (for strong convolutional baselines), and a small ViT-Tiny (for transformer-based behavior). This diversity allows us to study how MSVP interacts with very different architectural biases. Visual Prompting and Parameter-Efficient Tuning : Visual prompting draws inspiration from prefix tuning and prompt tuning in NLP [ 24 , 23 ] , which demonstrated that lightweight prompt parameters can adapt frozen language models to

Read the complete text