OmViD: Omni-supervised active learning for video action detection
Aayush Rana, Akash Kumar, Vibhav Vineet, Yogesh S. Rawat · 2025
Read OmViD: Omni-supervised active learning for video action detection by Aayush Rana, Akash Kumar, Vibhav Vineet, Yogesh S. Rawat (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-13983.txt.
Introduction
Aayush Rana and co-authors cut the labeling cost of teaching machines to spot actions in video. Full spatio-temporal labels are slow and pricey to make. Real clips also vary: some need only a tag, others a full mask. Their 2025 system, OmViD, first guesses the cheapest label type each clip needs. Label types run from video tags and points up to boxes and pixel masks.
Pseudo-labels then grow from those light marks by way of 3D superpixels. The model trains on the grown labels as if they were hand-made. On two standard action sets, costs fell hard with little accuracy lost. The plan suits any archive where expert hours are the scarcest resource.
Catalog record
| CREATOR | Aayush Rana, Akash Kumar, Vibhav Vineet, Yogesh S. Rawat |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-13983
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmViD: Omni-supervised active learning for video action detection Aayush Rana, Akash Kumar, Vibhav Vineet, Yogesh S. Rawat Abstract Video action detection requires dense spatio-temporal annotations which are challenging as well as expensive to obtain. However, real-world videos often have varying level of difficulty and may not require equal level of annotations. In this paper we analyze the types of annotation appropriate for each sample and how it affects spatio-temporal video action detection. We focus on two different aspects affecting video action detection; 1) how to obtain varying level of annotations for videos, and 2) how to learn video action detection with different types of annotations. We study several annotation types including i) video level tags , ii) points iii) scribbles , iv) bounding box , and v) pixel level masks . First, we propose a simple active learning strategy which estimates appropriate types of annotations required for each video sample. Next, we propose a novel learning based spatio-temporal 3D-superpixel approach which generates pseudo-labels from different types of annotations and enables learning of video action detection from such annotations. We validate our approach on two different datasets, UCF101-24 and JHMDB-21, for video action detection, significantly reducing the annotation cost without significant drop in performance. 1 Introduction Figure 1 : OmViD outperforms prior weakly and semi-supervised video action detection methods with lower annotation cost. We report v-mAP/f-mAP @ 0.5 on UCF101 under varying annotation budgets. Bubble size indicates the number of annotated frames per method. Our omni-supervised approach achieves better performance by efficiently annotating more frames at reduced cost. Figure 2 : Various types of annotations used for omni-supervised video action detection training. We show two frames from a video sample where each video is annotated with different annotation type. Video action detection has been a challenging task as it requires the localization of where an action occurs spatially within a frame as well as when it occurs temporally across multiple frames [ 21 , 61 , 76 , 23 , 11 , 10 , 48 ] . Recent advances in video action detection has stemmed from large models trained on fully annotated data with bounding box and pixel-level masks [ 76 , 80 , 63 ] . Unlike video action classification task which can be trained with only class labels for the video [ 6 , 18 , 31 , 33 ] , action detection training requires most of the frames to be correctly annotated. This poses a problem for scaling to large datasets as the number of frames to annotate increases rapidly, limiting big datasets to subsample frames for annotation [ 17 , 75 ] . Lack of large dataset availability also limits the improvement in video action detection task. Therefore, it is important to improve video action detection training using low cost annotation alternatives. Most recent works using limited annotations for video action detection follow weakly and semi-supervised approach [ 43 , 70 , 2 , 8 ] . They utilize the ground-truth annotation for a fraction of the training data and generate pseudo-labels for the remaining data. They are often limited by having to rely on external off-the-shelf object detector and post-processing steps (non-maxima suppression, metric based matching) [ 44 , 79 , 12 ] . These steps require dataset specific parameter tuning for external detector and post processing, further adding complications to the overall training process. Furthermore, these approaches under-perform significantly compared to supervised training. Recent semi-supervised methods have closed the performance gap [ 30 , 22 ] but still require 20% of the dataset to have full clean annotations, requiring large annotation cost. In this work, we study how varying level of annotations can be used for video action detection to further reduce the labeling cost. We focus on two different aspects; 1) how to obtain varying level of annotation for videos, and 2) how to learn video action detection with varying level of supervision from mixed annotation types. We study several levels of annotations; 1) video level tag , 2) point , 3) scribble , 4) bounding box and 5) pixel level mask . Bounding box and pixel-level mask incurs high annotation cost but provides high quality ground truth which is very effective for model training. Annotations such as scribbles, points and video tags on the other hand induce noise in training due to lack of ground truth information. First, we propose a simple active learning (AL) strategy to estimate the most appropriate annotation for each video which maximizes information gain while using lower annotation cost. Next, we utilize a learnable novel spatio-temporal deep 3D-superpixel module to obtain pseudo-labels. This enables an effective training of action detection models using multiple annotation types at the same time without relying on any external detectors or heuristic based post processing steps. To summarize, we make the following contributions; 1) We propose a unified Om ni-Supervised Vi deo Action D etection method ( OmViD ) which can utilize different types of annotations at the same time to train a video action detection model. This approach allows use of sparse annotations from varying categories, enabling annotation of more videos sparsely instead of having fewer videos with dense per frame annotation for equivalent annotation budget. 2) We propose joint learning of a unified pseudo-label estimator together with the action detection for video models, thereby removing reliance on external detector altogether. A novel learning based 3D superpixel module is used as pseudo-label estimator which predicts spatio-temporal superpixels which are integrated with low cost annotations to estimate the pseudo-labels. 3) We show that a simple and novel bucket-based active learning strategy for omni-label selection improves performance while keeping the annotation cost low. We perform our experiments on UCF101-24 and JHMDB-21 datasets to validate our approach. As shown in Figure 1 , we are able to obtain comparable performance with fully-supervised approach using 6% training data and using fewer annotation cost ( 6% @ 171 man hours ) compared to prior methods ( 5% @ 234 man hours, 20% @ 937 man hours ). 2 Related work Action detection: Spatio-temporal action detection in videos is a challenging task where the goal is to detect actions in both space and time for a given video [ 25 , 47 , 21 ] . Prior works utilize the fully supervised action detection approach using dense spatio-temporal annotation (bounding boxes, pixel-wise masks) to train the detection model [ 76 , 39 , 11 ] . A common strategy is to use 2D [ 62 , 19 , 58 ] or 3D [ 6 , 18 , 45 ] encoders to get video features and use external region proposals [ 15 , 51 ] for an accurate spatio-temporal action detection in multiple stages [ 76 , 21 ] . Alternatively, recent works have shown a single stage anchor-free approach without external proposals also perform competitively [ 39 , 11 ] . Either approach assumes dense annotations present for training, which is a costly in videos for spatio-temporal detection. Weakly/semi-supervised learning: Weakly [ 26 , 59 , 2 , 32 , 14 ] and semi-supervised [ 29 , 53 , 60 , 4 , 49 , 34 ] methods leverage limited annotations via consistency regularization [ 22 , 4