An empirical study of the effect of video encoders on Temporal Video Grounding

Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Felipe Bravo-Marquez, Department of Computer Science, University of Chile, CENIA and IMFD, Australian Institute for Machine Learning, University of Adelaide, National Institute of Advanced Industrial Science and Technology ignac · 2025

Read An empirical study of the effect of video encoders on Temporal Video Grounding by Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Felipe Bravo-Marquez, Department of Computer Science, University of Chile, CENIA and IMFD, Australian Institute for Machine Learning, University of Adelaide, National Institute of Advanced Industrial Science and Technology ignac (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17007.txt.

Introduction

Ignacio M. De la Jara and three co-authors test video feature choices in a 2025 paper. Temporal video grounding finds the moment matching a spoken query. A user describes an action, and the system locates it in video. Most research reuses only a few standard video encoders.

An encoder converts raw video into compact feature numbers. The team compared encoders based on three design families. Those families were convolutional networks, temporal reasoning models, and transformers. They tested on three well-known benchmark collections of videos. Simply swapping the encoder changed performance significantly. Clear error patterns tied to specific features also appeared. Different features seemed to complement each other. The study warns the field against overfitting to one favorite encoder.

Catalog record

CREATORIgnacio M. De la Jara, Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Felipe Bravo-Marquez, Department of Computer Science, University of Chile, CENIA and IMFD, Australian Institute for Machine Learning, University of Adelaide, National Institute of Advanced Industrial Science and Technology ignac
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17007

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An empirical study of the effect of video encoders on Temporal Video Grounding Ignacio M. De la Jara, Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Felipe Bravo-Marquez, Department of Computer Science, University of Chile, CENIA and IMFD, Australian Institute for Machine Learning, University of Adelaide, National Institute of Advanced Industrial Science and Technology [email protected], [email protected]@weblab.t.u-tokyo.ac.jp, [email protected] Abstract Temporal video grounding is a fundamental task in computer vision, aiming to localize a natural language query in a long, untrimmed video. It has a key role in the scientific community, in part due to the large amount of video generated every day. Although we find extensive work in this task, we note that research remains focused on a small selection of video representations, which may lead to architectural overfitting in the long run. To address this issue, we propose an empirical study to investigate the impact of different video features on a classical architecture. We extract features for three well-known benchmarks, Charades-STA, ActivityNet-Captions and YouCookII, using video encoders based on CNNs, temporal reasoning and transformers. Our results show significant differences in the performance of our model by simply changing the video encoder, while also revealing clear patterns and errors derived from the use of certain features, ultimately indicating potential feature complementarity. 1 Introduction Understanding and reasoning about long, untrimmed videos is at the core of Computer Vision (CV). As humans have the ability to intuitively identify relevant moments within videos, the task of Temporal Video Grounding (TVG) appears as a fundamental effort in CV, aiming to develop models that recognise and determine temporal boundaries of action instances in videos [ 28 , 12 , 10 ] using natural language queries [ 7 , 13 ] . As such, work on the TVG task is extensive and includes a wide variety of approaches and techniques. While the original models were mostly suggestion-based, more recent techniques have aimed at predicting the start and end temporal positions directly, or by regressing them from the input video. The recent advent of transformer-based models [ 32 ] has also brought new developments, where the integration of pre-training stages or the direct addition of pre-trained vision and speech models has led to significant performance improvements. Despite the large amount of prior work in the TVG task, we find that the role of the video representation has not been consistently investigated so far. Specifically, we observe that prior work has relied on features derived from action classification models such as C3D [ 30 ] or I3D [ 2 ] , with alternatives remaining relatively unexplored. We speculate that this selection bias may lead to model designs that overfit or exploit spurious patterns in these features, without generalising in the long run. We suggest that the increased complexity of recent approaches in search of improved performance can be seen as indirect evidence of this point. In light of this issue, in this paper we propose a comprehensive empirical study to shed light on the role of video representation in the TVG task. We consider three relevant benchmark datasets, Charades-STA, ActivityNet Captions, and YoucookII, and design a comprehensive framework that allows us to isolate the effect of different video representations by introducing minimal changes in the model architecture. We use a wide variety of pre-trained models to extract video representations, including more than 10 different types of models, resulting in more than 30 sets of extracted features for our data. These include, but are not limited to, well-known CNN and Transformer-based action classifiers. We release features to encourage research in this area. 1 1 1 https://github.com/Mezosky/VideoFeatures_TVG Our results show that changes in the video representation can lead to substantial performance improvements by keeping the models as is, allowing a “classical” approach [ 25 ] to perform better by a large margin by simply changing the video encoder, findings that are consistent with similar observations recently made in the context of query representations [ 16 ] . Our experiments reveal clear pitfalls in the use of certain features and uncover complementarity between features, which could lead to further performance improvements if exploited. 2 Related Work Action Classification The task of identifying the action being performed in a video is the cornerstone of current video understanding pipelines. It is essential for the task of temporal video grounding, as it is the current way of encoding videos. Current methods are usually trained via supervised learning, using datasets such as Something-Something [ 11 ] , Kinetics [ 2 ] , among others, which contain short video clips of a single action. Different methods have been proposed to solve the action classification task, the most recent approach based on neural networks could be divided into CNN-based [ 31 , 2 , 6 , 5 ] , temporal reasoning [ 19 , 33 ] and transformers [ 4 , 18 ] . In the case of CNN-based, the methods exploit the ability of CNNs to capture spatial information in images and add various techniques to extract the temporal information from the video. [ 31 ] proposed a convolutional block, which is the combination of 2D convolutions followed by a 1D convolution. Such blocks approximate the behaviour of 3D convolutions and allow to capture the temporality from the videos. Knowing that the information in a video is very redundant and not symmetric, [ 6 ] proposed to use a slow and a fast track, aiming to capture the spatial and temporal information respectively. Temporal reasoning classifiers aim to capture the temporal information between features extracted from frames using techniques that exploit latent information in videos, such as redundancy (sampling), summarisation (aggregation), and ordering (ranking). Limin et al. [ 33 ] use a sampling technique to select features that, after passing through an aggregation mechanism, effectively represent the video. Aiming to create a better representation of the videos, Lin et al. [ 19 ] proposed a shift module that applies a shift in part of the temporal channel of the features to exchange information between frames. The advent of transformer-based models has also led to new developments in this task [ 4 , 1 ] . However, their success in this task is due to their flexibility in capturing spatial information and their inherent ability to model sequential information. [ 4 ] presents a multiscale feature hierarchy that combines low-level spatial information processed at early layers with more complex high-level features that capture temporal information processed at deeper layers. Temporal Video Grounding The task that concerns this work, where we find mainly two types of approaches. On the one hand, we find proposal-based models that, given a query, return a set of candidates that can later be ranked [ 20 , 8 , 3 , 36 , 38 ] . Recent approaches include both transformer-based models and contrastive losses [ 24 , 27 , 35 , 37 , 39 ] . On the other hand, we find proposal-free models that try to predict the start and end locations directly from the video span [ 9 , 25 ] . 3 Experimental Framework Model At the core of our experimental framework lies the TVG model to be used as a pivot for testing video features. Although there are a variety of approaches to

Read the complete text