ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting
Clayton Fields, Casey Kennington · 2026
Read ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting by Clayton Fields, Casey Kennington (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18452.txt.
Introduction
Fields and Kennington train compact vision-language transformers for low-resource settings with ESsEN. Most vision-language models carry tens of billions of parameters, too many for edge devices or independent robots. Inspired by child language learning under sparse data, the authors show two-tower encoders beat one-tower encoders on discriminative English tasks. Adding convolutional networks aids parameter efficiency, and the fusion module tolerates varied shapes and sizes at equal performance, making the field more accessible.
Catalog record
| CREATOR | Clayton Fields, Casey Kennington |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36102 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18452
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ESsEN: Training Compact Discriminative Vision-Language Transformers in a Low-Resource Setting Clayton Fields, Casey Kennington Abstract Vision-language modeling is rapidly increasing in popularity with an ever expanding list of available models. In most cases, these vision-language models have parameters in the tens of billions, which is necessary for some needs, but in many cases smaller models are necessary (e.g., on edge devices or independent robotic platforms). Unfortunately, there is little research in producing light-weight models or in training them with small datasets. Inspired by the language learning progression and data sparsity in child development, in this paper, we address both of these goals in a systematic fashion. We show that two-tower encoder models are superior to one-tower encoders in low-resource settings for discriminative English tasks. We show also that incorporating traditional convolutional networks into the two-tower transformer architecture can help produce parameter efficient vision-language models. Finally, we show that the cross-modal fusion module of two-tower encoders can vary significantly in shape and size while producing the same results. In addition, we present ESsEN, a compact vision-language model that can be trained end-to-end with relatively few resources that performs as well on several tasks with only a fraction of the parameters compared to other models. The experimental results and the tools we present here make vision-language modeling more accessible to a wider variety of researchers. 1 Introduction The adaptation of the transformer model 32 to vision-language (VL) tasks has revolutionized the field of VL modeling. In the space of only a few years, a huge array of multimodal (mostly vision) transformers have appeared in the literature. One thing that all of these models have in common is that their high parameter size translates to increasingly large compute and training data requirements to train them from end-to-end. At the smaller end are early encoder-only models such as VisualBERT, which contain around 100M parameters. This problem has become acute in recent years, as extremely large, commercial models require compute and electricity to function that make them commercially non-viable. Additionally, these "small" models are generally trained with very large datasets, often consisting of millions of unique images paired with text (Conceptual Captions 27 for instance) that are difficult to create, process and store. Furthermore, datasets are usually scraped from internet, often with text derived from ALT text captions, which may not translate to accurate depictions of the images. Datasets derived from human annotators, such as MSCOCO 20 , or with captions in low resource languages are naturally much smaller and more difficult to develop. In conjunction with practical constraints are considerations from child development. American children begin speaking, on average, at around 12 months of age 13 with some estimates that children only need to hear about 5 million words before they can utter recognizable words. While it is difficult to estimate how much visual stimuli children have been exposed to and how much of a child’s visual experience is integrated into the linguistic system, it is not controversial to claim that the meaning of many words is ’grounded’ into embodied experience, particularly visual experience 14 . For example, color words or animal words refer to visually perceived entities. Visual language models are a step in the direction of bringing the visual and linguistic worlds together, but work needs to be done to make compact VL models that train and operate efficiently on developmentally plausible sized datasets. Creating compact and data-efficient transformer models trained on text has been an active area of research. Models such as DistilBERT 25 , TinyBERT 17 and MobileBERT 29 are greatly reduced in size and compute requirements. And studies such as 8 have shown that transformers can be both parameter and data efficient by simply training them from end-to-end. Unfortunately, there have to date been very few of these efforts with respect to VL transformers. The few studies that have so far been produced have primarily concentrated on using knowledge distillation, a process whereby the output of a large "teacher" model is used to train a smaller "student" model 34 . Inspired by distilled transformers such as DistilBERT 25 , training models of this nature is both logistically and conceptually very complex. Other efforts have adopted the strategy of pruning, removing some proportion of the weights in a large pretrained model 11 . Both of these strategies rely however, on incorporating a full-size pretrained model at some point in the training process. For many researchers and practitioner of VL models, this is not desirable and confound a model’s results when end-to-end training is indicated. In this study, we seek to address these gaps by investigating the training and evaluation of small-scale models for discriminative (i.e,. not generative) VL tasks in a low-resource setting. The goals of this study are two-fold. First, we hope to establish that small-scale transformers can perform well on VL tasks with small models, small training datasets, and limited compute. Secondly, we create and present a compact model resulting from these efforts that can be used for research or practical applications by researchers with limited resources. Our specific contributions can be summarized as follows: • We show that for compact VL transformers, with limited training data and compute, two-tower encoder models are more effective than one-tower encoders. 1 1 1 It is fairly common to refer to towers as streams (e.g., a visual and a text stream), though in the literature both terms are used. We follow 9 and use the term tower in this paper. • We show that we can create efficient VL models by incorporating convolutional vision models into a two-tower encoder architecture. • We show that a variety of configurations to the cross-modal fusion layers of a two-tower encoder lead to the same downstream results. • We present a compact VL model, using the convolutional image model EfficientNet-b2 30 , that can be trained end-to-end with relatively low compute and data inputs. The rest of this paper will proceed as follows: Section 2 cover some of the relevant related work. Section 3 will cover the datasets used for both for pretraining and evaluation. In Section 4 we determine whether a one-tower or a two-tower encoder is best suited for parameter-efficient modeling. Section 5 describes our experiments for determining the best backbone architecture for our small-scale VL transformer. The third and final set of experiments, described in Section 6 , we investigate the other architectural elements of our compact VL transformer and discuss the results of our final model in Section 7 . Finally we close the paper with some brief concluding remarks in Section 8 . 2 Related Work Though the field of VL modeling has expanded rapidly in recent years (see 9 for an overview), there has to date been relatively little research on parameter-efficient VL modeling. As in the domain of natural language processing, most research has focused on compressing existing pretrained models using techniques such as knowledge distillation or pruning. DistilVLM 7 and EfficientVLM 34 both use knowledge distillation much the way that models like DistilBERT 25 do. Studies such as 11 on the other hand, attempt to compress VL transformers through pruning (i.e., removing parameters) while others use quantization techniques. Works such as the MiniVLM model introduced in 33