Leveraging Vision–Language Pre - training for Human Activity Recognition in Still Images
Cristina Mahanta, Gagan Bhatia · 2025
Read Leveraging Vision–Language Pre - training for Human Activity Recognition in Still Images by Cristina Mahanta, Gagan Bhatia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13458.txt.
Introduction
Cristina Mahanta and Gagan Bhatia wrote a 2025 computer vision paper on recognizing human activity in single still photos. A still image lacks motion cues, which makes actions like running versus standing hard to tell apart. Yet still image recognition would serve photo indexing, safety monitoring, and assistive tools.
The authors labeled 285 images from the MSCOCO collection into four classes: walking, running, sitting, and standing. A convolutional network trained from scratch reached only forty one percent accuracy. Fine tuning CLIP, a model pretrained to match images with text, lifted accuracy to seventy six percent. The jump shows that vision language pretraining transfers powerfully to this small data task. The study is small, but the contrast is stark.
Catalog record
| CREATOR | Cristina Mahanta, Gagan Bhatia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-13458
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Leveraging Vision–Language Pre - training for Human Activity Recognition in Still Images Cristina Mahanta, Gagan Bhatia Abstract Recognising human activity in a single photo enables indexing, safety and assistive applications, yet lacks motion cues. Using 285 MS - COCO images labelled walking/running, sitting and standing, scratch CNNs scored 41% accuracy. Fine - tuning multimodal CLIP raised this to 76%, proving contrastive vision - language pre - training decisively improves still - image action recognition in real deployments. 1 Introduction Detecting human activities from still images is a challenging problem in computer vision, largely due to the subtle and complex variations inherent in human behaviour. In this work, we address this task using a subset of MSCOCO 2017 validation split introduced by Lin et al. 2015 , each labeled as walking/running, sitting, or standing. We begin with two baseline models — Convolutional Neural Network (CNN) and Feedforward Neural Network (FNN) — and then enhance performance through data augmentations, dropout, weight decay, and early stopping. To utilize broader visual knowledge, we apply transfer learning with pretrained Vision Transformers and contrastive models (e.g., CLIP) and explore multimodal embeddings for richer feature representations. We detail the preprocessing steps, model configurations, and evaluation metrics to enable transparent comparison across all methods. Figure 1: Examples of the training data 2 Description of Data and Methods 2.1 Data The dataset for this study is drawn from a carefully curated subset of the Microsoft COCO (Common Objects in Context) validation split, originally introduced by Lin et al. 2015 and now a gold standard benchmark in computer vision tasks such as object detection, segmentation and image captioning. From the full COCO set, we selected 285 images depicting exactly one of the three human activities—walking/running (98 images), sitting (95 images), or standing (92 images) —yielding a nearly balanced three-way classification problem. All images were downloaded directly via their URLS, and none were discarded due to corruption or missing annotations, confirming complete data integrity. A few examples of the training dataset are shown in Figure 1 . (a) Scatter of width vs. height (b) Box - plots by activity (c) Aspect ratio histogram (d) Overlaid height/width distributions Figure 2: Exploratory data analysis of image dimensions: (a) width vs. height scatter, (b) height/width box - plots grouped by activity, (c) distribution of aspect ratios, and (d) overlaid height and width histograms. Label Count % of Total Width (mean \pm \sigma , px) Height (mean \pm \sigma , px) Aspect Ratio (mean \pm \sigma ) walking_running 98 34.4% 576.1\pm 60.2 492.3\pm 85.7 1.17\pm 0.21 sitting 95 33.3% 558.2\pm 68.5 495.8\pm 95.4 1.19\pm 0.23 standing 92 32.3% 558.7\pm 64.1 512.2\pm 87.2 1.22\pm 0.19 Overall 285 100.0% 565.7\pm 99.2 499.4\pm 100.5 1.20\pm 0.22 Table 1: Dataset and per-class image statistics. Ranges (width: 300–640 px; height: 240–640 px) apply across all classes. No missing or corrupted entries were found. After performing a detailed exploratory data analysis (EDA), it can be seen that the images range from 300 to 640 pixels in width, clustering around an average of approximately 566 pixels, and from 240 to 640 pixels in height, centred near 499 pixels, as shown in Table 1 . Moreover, when comparing the distribution plots for walking/running, sitting, and standing, their medians, interquartile ranges, and overall distributions align almost perfectly. This indicates that no activity label consistently contains larger or smaller images. Table 1 also highlights that the class frequencies remain nearly identical, with each activity accounting for roughly one-third of the dataset, so no label imbalance is expected to bias model training. The scatter and box plots in Figure 2 further confirm that no class systematically contains larger or smaller images. At the same time, the aspect-ratio histogram exhibits two dominant modes around 1.0 (square image) and 1.33 (4:3), with an overall mean ratio of approximately 1.2. Together, these findings demonstrate that the dataset is inherently balanced and free of resolution- or framing-based biases. This means that any necessary standardisation (e.g. image resizing and normalisation) can be applied uniformly, and the augmentation strategies need to focus only on semantic diversity rather than correcting for class-specific size or aspect ratio artefacts. 2.2 Models In this work, we compare several neural network architectures and transfer learning approaches for image classification. In the following subsections, we describe each model and the underlying design choices. CNN and FNN The CNN_base model follows a classic convolutional design: three blocks of 3\times 3 convolutions (padding=1) each succeeded by ReLU activations and 2\times 2 max-pooling, with the resulting feature map flattened into a two-layer fully connected classifier. By exploiting local spatial correlations and hierarchical feature extraction, it embodies the core principles of convolutional networks Lecun et al. 1998 ; O’Shea and Nash 2015 and serves as a robust baseline. In contrast, the FNN_base model treats each image as a flat vector passed through successive dense layers with nonlinearities. Lacking the spatial inductive biases and weight sharing of convolutions, this fully connected architecture consistently underperforms on visual data LeCun et al. 2015 . Generalising CNN The CNN_gen model extends the baseline CNN architecture by integrating several regularization and normalization techniques to improve generalization. Batch normalization Ioffe and Szegedy 2015 is applied after each convolution to stabilize activations, and dropout layers are interleaved with the convolutional and fully-connected blocks to prevent overfitting. These modifications, combined with data augmentation during training, enable the CNN_gen model to achieve better performance on unseen data while maintaining the simplicity of convolutional feature extraction. Transfer Learning for Binary Classification We leverage a pretrained Vision Transformer (ViT) backbone, which splits each image into 16\times 16 patches and processes the resulting sequence through standard transformer blocks to capture long-range dependencies and global context Dosovitskiy et al. 2021 . Transformers’ self-attention mechanism and large-scale pretraining yield highly generalizable feature representations, making them particularly well suited for transfer learning across diverse vision tasks. In addition, we fine-tune two modern vision–language encoders — CLIP Radford et al. 2021 and SigLIP2 Tschannen et al. 2025 — by replacing their projection heads with a two-way classifier and optimizing all parameters end-to-end under a cross-entropy objective. Transfer Learning for Multiclass Classification For multiclass tasks, we adopt the same pretrained ViT, CLIP, and SigLIP2 models, extending each classification head to output C logits, where C is the number of target categories. All three backbones are fine-tuned jointly with the new head under cross-entropy loss, allowing them to adapt their rich, pretrained representations to the specific demands of our domain-specific multiclass classification problem. CLIP Image Embeddings We use CLIP embeddings in two complementary ways. In the first setting, we treat CLIP Radford et al. 2021 as a pure image encoder: given a image we encode it to obtain a fixed - length vector of dimension d (typically 512). These image vectors are paired with their ground - truth labels to build a simple PyTorch dataset, which we split into training, validation, and test subsets. We then train a small multilayer perceptron ( MultimodalClassifier ) on top of the raw CLIP embeddings, consisting of several fully connected layers with ReLU, batch normalization, and dropout, and optimize with cross - entropy loss. This setup tests how linearly separable the CLIP image representations are for our target classes. CLIP Image-Text Embeddings In the