SuperCap: Multi-resolution Superpixel-based Image Captioning
Henry Senior, Luca Rossi, Gregory Slabaugh, Shanxin Yuan · 2025
Read SuperCap: Multi-resolution Superpixel-based Image Captioning by Henry Senior, Luca Rossi, Gregory Slabaugh, Shanxin Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08496.txt.
Introduction
Image captioning systems often lean on object detectors to find things in a picture before writing a sentence about it. Henry Senior and three co-authors present SuperCap in 2025, a system that swaps detectors for superpixels, small patches that group nearby pixels of like color and shape. A vision language model then reads those patches, so the system can name things it was never trained to detect.
SuperCap also views each image at several levels of detail at once. An attention step weighs which level matters most for the words being written. Tested on the COCO data set, the full system reached a CIDEr score of 136.9, a mark the authors call competitive. Tests that removed parts one at a time showed what each part adds.
Catalog record
| CREATOR | Henry Senior, Luca Rossi, Gregory Slabaugh, Shanxin Yuan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08496
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SuperCap: Multi-resolution Superpixel-based Image Captioning Henry Senior, Luca Rossi, Gregory Slabaugh, Shanxin Yuan Abstract It has been a longstanding goal within image captioning to move beyond a dependence on object detection. We investigate using superpixels coupled with Vision Language Models (VLMs) to bridge the gap between detector-based captioning architectures and those that solely pretrain on large datasets. Our novel superpixel approach ensures that the model receives object-like features whilst the use of VLMs provides our model with open set object understanding. Furthermore, we extend our architecture to make use of multi-resolution inputs, allowing our model to view images in different levels of detail, and use an attention mechanism to determine which parts are most relevant to the caption. We demonstrate our model’s performance with multiple VLMs and through a range of ablations detailing the impact of different architectural choices. Our full model achieves a competitive CIDEr score of 136.9 136.9 on the COCO Karpathy split. 1 Introduction Image captioning, the challenging task of developing a model that describes a visual scene in natural language, has received considerable attention in recent years from the computer vision community. It is a task with great scientific impact ( e.g . , ensuring models can truly describe the scene) and societal impact ( e.g . , captioning models provide benefits in terms of accessibility through generating alt-text for websites). While early works focused on a standard Encoder-Decoder architecture [ 16 ] using a Convolutional Neural Network (CNN) to produce a feature vector that is decoded into a caption via a Recurrent Neural Network (RNN), later works moved to adopt object detectors. There are typically two approaches to image captioning: detector-based (Bottom Up Top Down (BUTD) [ 4 ] and its descendants [ 7 , 44 , 43 ] ) or ‘detector-free’ models, the latter of which are mostly massively pretrained large Vision-Language Models (VLMs) built on top of the Transformer architecture [ 34 ] and pretrained on large datasets such as LAION-5B [ 31 ] . These models produce strong global features but risk missing out on more nuanced finer grain detail, something at which detector-based methods naturally excel. In this work, we investigate taking the best of both worlds, answering the question, ‘how can we leverage VLMs to produce high quality captions whilst utilising finer object details?’. We achieve this through the use of superpixels: by segmenting the image into M superpixels, we leverage the zero shot classification capabilities of VLMs to replace the object detector. To ensure finer-details are captured, we incorporate a multi-resolution approach, where the image is divided into a set of superpixels of different resolutions. We also make use of a whole image feature to maintain the global context. Early deep learning approaches to image captioning focused on using a CNN to represent the entire image using a unique latent feature. This feature was then decoded into a caption using an RNN. Although this approach was initially successful, it produced captioning models that would focus on the global context of an image and would therefore miss specific objects [ 16 ] . As object detectors improved, they began to be used in image captioning models to address the weaknesses of earlier approaches. Following on from Anderson et al. ’s BUTD [ 4 ] work, a large number of image captioning architectures based on object detectors have been produced [ 7 , 44 , 43 ] . Whilst these models all provide strong performance and novel ways of utilising the objects in different ways, they all share a common flaw - they are fundamentally limited by their object detectors. Specifically, 1) the closed set nature of object detectors, which results in only objects in the training data of the detector being identified, and 2) the expensive computation required by object detectors (FasterRCNN [ 29 ] with a ResNet-50 [ 11 ] takes 447 GFLOPs). As object detectors are only as good as their training data, in scenarios where the detector misclassifies an object (or is unable to classify it), the caption will be incorrect. The closed-set nature of detector-based image captioning, alongside the development of large pretrained VLMs has led researchers to develop detector-free captioning architectures [ 36 ] . These approaches largely address the problems of detector-based architectures by scaling the Transformer and the data used to train the model. Given the impressive Transformer scaling laws [ 15 ] , the development in computing infrastructure, and the increasing accessibility of ever larger datasets, this approach has proven to be the dominant approach in recent years. Methods aiming to unify detector-based and detector-free are limited [ 10 ] . Their architecture mirrors that of the detector-free models [ 36 ] , a ViT combined with a transformer language model. However, they include a novel ‘concept token network’ that predicts concepts from the hidden layers of the ViT, essentially providing object detection via proxy. This approach leads to a complex model that is difficult to train. In comparison, our model is easily trainable end-to-end following standard procedures for an image captioning model trained on COCO [ 24 , 6 ] . In this paper we present SuperCap, a novel detector-free image captioning architecture that joins together the benefits of detector-based and detector-free approaches to image captioning. The key contributions of our paper are as follows: 1. SuperCap is the first image captioning model to be based on superpixels instead of patches or object bounding boxes. Our model performs competitively on standard benchmarks whilst having an architecture that scales appropriately to future developments within the computer vision community. 2. To the best of our knowledge, we are the first image captioning architecture to make explicit use of multiple resolutions, thus ensuring that our model captures the local context (like detector-based models) and the global context (like pretrained detector-free models). We empirically show the advantage of our multi-resolution approach through extensive ablation studies. The remainder of this paper is organised as follows. Section 2 reviews the related work, while Sections 3 introduces SuperCap. In Section 4 we discuss the implementation details of our model as well as of the training and evaluation protocol. We then conduct an extensive experimental evaluation in Section 5 and we conclude the paper in Section 6 . 2 Related Work In this section we detail the related work and background to provide context for the literature that works sits alongside. We start by giving an overview of image captioning, followed by detector-free approaches to this problem. The section concludes with a brief overview of superpixels. (a) Bounding Boxes One common way in which regions are extracted is through the use of bounding boxes. This is used by Anderson et al. [ 4 ] and subsequent works [ 7 , 40 , 44 ] . (b) Patches A more contemporary approach to region extraction is through the use of patches [ 10 ] . However, it results in objects being divided between features, as shown above. (c) Superpixels Our approach utilises superpixels to create regions that encompass the whole image but ensure that objects are not unnecessarily divided between features. Figure 1 : Visual Representation of Different Region Extraction Techniques. Here we show the three different ways in which