In Context Learning with Vision Transformers: Case Study

Antony Zhao, Alex Proshkin, Fergal Hennessy, Francesco Crivelli · 2025

Read In Context Learning with Vision Transformers: Case Study by Antony Zhao, Alex Proshkin, Fergal Hennessy, Francesco Crivelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20872.txt.

Introduction

Antony Zhao and three co-authors test in-context learning in vision transformers in this 2025 paper. Language transformers can learn a task from examples inside the prompt, with no retraining. They can even pick up whole function classes, like lines or small networks, from random data. The authors ask whether vision models can do the same in image space. Their case study feeds example images and a query, then checks whether the model infers the hidden rule. Target rules include convolutional networks and other image methods.

The study is framed as a first probe rather than a verdict. Image rules are richer than the toy functions used in language tests. Success would mean a vision model could mimic a chosen algorithm from demonstrations alone. Failure would map where the analogy between text and image learning breaks.

Catalog record

CREATORAntony Zhao, Alex Proshkin, Fergal Hennessy, Francesco Crivelli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20872

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In Context Learning with Vision Transformers: Case Study Antony Zhao, Alex Proshkin, Fergal Hennessy, Francesco Crivelli Abstract Large transformer models have been shown to be capable of performing in-context learning. By using examples in a prompt as well as a query, they are capable of performing tasks such as few-shot, one-shot, or zero-shot learning to output the corresponding answer to this query. One area of interest to us is that these transformer models have been shown to be capable of learning the general class of certain functions, such as linear functions and small 2-layer neural networks, on random data (Garg et al, 2023). We aim to extend this to the image space to analyze their capability to in-context learn more complex functions on the image space, such as convolutional neural networks and other methods. Introduction In-context learning is a capability of large transformer models, such as some large language models like ChatGPT. It allows the model to learn from the samples in the prompt to predict some output without having to train on these samples directly. garg2023 showed that this performance can be comparable to an almost optimal solver, such as comparing its ability to learn the general class of linear functions against the performance of a least squares algorithm. We aim to extend this to the image space to further investigate and increase our understanding of in-context learning. Recently, Vision Transformers have been shown to be noteworthy for their small inductive bias. They have exceeded conventional Convolutional Neural Networks on many benchmarks, but they often struggle to learn meaningful representations on small datasets. In-Context Learning has been shown to allow transformers to infer latent concepts and patterns at test time, significantly improving accuracy. Notably, it has been shown by Min, Lyu, Holtzman et al. ( ArXiv ) that providing a strong pattern of input-output context significantly improves test results roughly the same amount, regardless of whether the context mappings match ground truth. In this paper, we apply the techniques from in-context learning to vision transformers (ViT) and assess their performance, in comparison to convolutional networks trained on the same data, allowing us to analyze the potential to use these models on smaller datasets by utilizing ICL. Related Work CNN vs ViT: Bai et al. (2021) conduct a rigorous evaluation of the robustness of Vision Transformers (ViTs) compared to Convolutional Neural Networks (CNNs) under standardized training conditions. Their findings indicate that CNNs can achieve adversarial robustness on par with ViTs when trained using similar methodologies. Additionally, while ViTs exhibit superior generalization to out-of-distribution data, this advantage is primarily attributed to their self-attention mechanisms rather than pretraining on large-scale datasets. This study underscores the importance of fair experimental setups in assessing model robustness and provides insights into the architectural factors influencing performance. ArXiv ViT Transfer Learning vs Training from Scratch: Steiner et al. (2021) perform a comprehensive empirical study of Vision Transformers (ViT), isolating how training-data scale, data-augmentation/regularization ("AugReg"), model size and compute budget interact/ By training more than 50 000 ViT variants on ImageNet-21k and JFT-300M under a unified protocol, they show that carefully tuned AugReg combined with additional compute can recover the accuracy gains normally obtained by a 10 × larger pre-training dataset, and that these gains transfer to downstream tasks after fine-tuning. The study also quantifies compute–performance trade-offs and releases all checkpoints, providing practical guidelines for data- and compute-efficient ViT training. ArXiv Are Transformers More Robust Than CNNs? Bai et al. (2021) conduct the first rigorously controlled comparison of model robustness between Vision Transformers and CNNs by matching model capacity (DeiT-S vs. ResNet-50) and unifying all training hyperparameters, optimizers, and data-augmentation schedules. They show that when CNNs inherit the strong augmentation and AdamW/weight-decay "Transformer" recipe, their adversarial robustness (PGD, AutoAttack, texture-patch attacks) becomes indistinguishable from that of Transformers, overturning earlier claims based on mismatched experimental setups. In contrast, Transformers still exhibit superior out-of-distribution generalization on ImageNet-C, ImageNet-A and Stylized-ImageNet; ablations attribute this benefit to the self-attention architecture itself rather than to large-scale external pre-training or other training heuristics. The study also introduces an augmentation-warm-up curriculum that stabilizes adversarial training for ViTs and releases complete code and checkpoints, establishing a reproducible baseline for future robustness research. ArXiv Few-Shot Vision Transformers (ICL) Dong et al. (2022) revisit few-shot visual classification with Vision Transformers and identify that, unlike CNNs, vanilla ViTs struggle to learn reliable patch-token dependencies when only a handful of labelled examples are available. They introduce Self-promoted sUpervisioN (SUN), which first self-pretrains a ViT on the target few-shot dataset and then uses the resulting teacher to create location-specific similarity labels that act as additional patch-level supervision during meta-training. Two refinements—background-patch filtration and spatial-consistent augmentation—further enhance the quality of the generated local labels. Evaluated with Meta-Baseline on mini-ImageNet, tiered-ImageNet and CIFAR-FS (5-way, 1-/5-shot), SUN lifts ViT accuracy above previous ViT baselines and, for the first time, surpasses strong CNN counterparts, demonstrating that appropriate token-wise guidance can compensate for Transformers’ lack of inductive bias in low-data regimes. ArXiv Problem Statement Extending the findings of garg2023 , our goal is to extend in-context learning performance gains to the image space. Using input-output pairs sampled from the dataset and sample functions, we generate a series of images and embeddings and pass them into our Transformer model as context for our final output. We investigate various models for our Transformer over different function classes, such as randomized convolutional and ViT models. Methods We first construct a transformer model that is capable of handling both images and embeddings. For the prompt, we pass the sequence of (x_{1},f(x_{1}),x_{2},f(x_{2}),...x_{k},f(x_{k}) , where f(\cdot) is a sample function from some function class, such as linear or convolutions, to our model. This model is then trained to predict the f(x_{i}) sequence, while masking the following values to ensure it only has the preceding context as information for each prediction. We process images by passing them through a ViT or CNN to obtain image embeddings. We then pass these image embeddings alongside the f(x_{i}) embeddings produced by the sample function or model into a decoder-only GPT-2 style model and obtain the predictions for each of the f(x_{i}) . These two models, (the ViT or CNN, and the GPT-2 model) are trained alongside each other, and are randomly initialized at the start. We sample x_{i} from the set of CIFAR-10 images, which we downscale from a 32\times 32 image to a 8\times 8 image. This is also normalized by subtracting the mean and standard deviation across the entire dataset. We chose to use these downscaled images rather than random noise as there are inherent patterns in the image that could be interesting to study. Figure 1: Example image from the CIFAR10 and the corresponding downscaled image. We use a variety of function classes for our sample functions. These include randomized linear functions, randomized convolutional networks, and randomized ViT using sinusoidal positional embeddings. Similar to garg2023 , we employ curriculum training by starting with a low-dimensional image by masking other values outside of a square d\times d block to 0 and slowly increasing the dimensionality of the image as well as the number of samples used in the prompt. Generally, we increment d by one each time and use n=kd+1 samples, where k

Read the complete text