Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation?

Luc P.J. Sträter, Hazel Doughty · 2026

Read Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation? by Luc P.J. Sträter, Hazel Doughty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03837.txt.

Introduction

Sträter and Doughty ask where temporal context is truly needed when adapting video models with little data. Parameter-efficient fine-tuning and probing adapt foundation models cheaply. Video versions usually adapt image-pretrained models. Standard methods can also run on video representations directly. The two settings are rarely compared head to head. Both also confine temporal reasoning to a single model part.

This study compares adaptation plans across settings in a systematic way. Appearance-focused, motion-focused, and spatially dense tasks are all covered. Low-data scenarios get special focus, since efficiency matters most there. How temporal context spreads across backbone, adapter, and probe is the central variable. Results show that allocation choice drives adaptation quality. Teams adapting video models on small budgets get direct guidance. Not every stage needs time context, and the study maps which do.

Catalog record

CREATORLuc P.J. Sträter, Hazel Doughty
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03837

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Where Do We (Not) Need Temporal Context in Low-Resource Video Task Adaptation? Luc P.J. Sträter, Hazel Doughty Abstract Parameter-efficient fine-tuning (PEFT) and probing enable adaptation of foundation models using only a small number of trainable parameters, making it attractive for video understanding where annotation and computation are expensive. However, video PEFT has focused on adapting image-pretrained models, while standard PEFT methods can also be applied to video representations. These settings are rarely compared and both confine temporal reasoning to a single component of the model, leaving open how temporal context should be distributed across backbone, PEFT and probe. In this work we provide a systematic study of model adaptation strategies for video understanding. We evaluate methods across appearance-focused, motion-focused and spatially dense settings, with a particular focus on scenarios with limited data where parameter-efficiency is most beneficial. Our results provide new insights into PEFT and probing across settings and demonstrate the importance of temporal context allocation for effective video adaptation. Project page: https://lucstrater.com/temporal-context/ . 1 Introduction Foundation models [ 7 ] have become the dominant paradigm in computer vision, offering generalization through internet-scale pretraining [ 50 , 55 , 2 , 9 ] . Despite learning transferable representations, these models still require adaptation to task-specific objectives, label spaces or domains. This is particularly costly for video, where both annotation and computation are significantly more expensive than for images. As a result, adaptation is increasingly performed through task-specific probing rather than full finetuning [ 3 , 12 ] . In parallel, parameter-efficient finetuning (PEFT) methods [ 23 , 17 ] adapt models by training only a small number of parameters [ 44 ] . Yet how these two paradigms interact, and how they should be applied in practice for video understanding, remains unexplored. Existing work in video PEFT has either adapted image-pretrained models to video [ 44 , 49 ] or evaluated standard PEFT on video-pretrained models [ 67 ] , without directly comparing the two. Beyond this gap, these works also make different assumptions about where temporal reasoning should occur. Methods built on image-pretrained backbones introduce temporal reasoning only in the adapter [ 44 , 49 ] , while those built on video-pretrained models limit it to the backbone [ 67 , 70 , 35 ] . In both cases, the temporal context is limited to one part of the model. Yet there is no reason this design should be optimal. Different tasks may benefit from temporal reasoning at different stages of the adaptation, and the cost of temporal context varies across components. In this paper, we present a systematic study of PEFT methods and probing strategies in video understanding. We focus on settings where model adaptation is most valuable: low-resource tasks from specialized domains where data or annotation is inherently limited. Our benchmark spans appearance-focused, motion-focused, and spatially dense tasks with diverse annotation requirements. To study temporal reasoning, we distribute temporal context across the backbone, probe, and PEFT modules, allowing each to operate on a different number of frames, and study the resulting performance-efficiency trade-offs. Our study reveals four key findings: (1) Standard PEFT methods ( e.g. LoRA [ 24 ] , AdaptFormer [ 11 ] ) outperform image-to-video PEFT approaches when paired with an attentive probe. This gap widens with a video-pretrained backbone. (2) On video backbones, attention-based adaptation excels on motion-heavy tasks, while MLP-based adaptation is best for spatial prediction. (3) Although probe performance differs substantially per task, this variability largely disappears once combined with PEFT, suggesting that a simple attentive probe is sufficient and removes the need for elaborate task-specific decoders. (4) Temporal modeling is driven by the backbone, which also dominates throughput. Temporal PEFT can partially compensate for reduced temporal context in the backbone, highlighting a promising direction for efficient video adaptation. In summary, we (i) provide the first comprehensive evaluation of parameter-efficient adaptation and probing strategies for video, covering both image-pretrained and video-pretrained backbones across diverse tasks and settings. (ii) Introduce temporal context distribution as a new perspective for understanding and improving parameter-efficient model adaptation in video. (iii) Offer concrete adaptation strategies for low-resource video tasks spanning appearance, motion, and spatial understanding. 2 Related Work Video Task Adaptation. Video understanding commonly relies on large pretrained encoders [ 1 , 6 , 58 , 63 , 62 , 64 , 3 , 9 , 2 , 10 , 61 ] , however adapting them to downstream tasks via end-to-end finetuning is computationally expensive and can overwrite useful priors [ 3 ] . This motivates probing strategies, where a lightweight prediction head is trained on a frozen encoder. Probe design varies in how temporal information is modeled. Linear probes aggregate tokens before classification, discarding temporal structure, while attentive probes [ 13 , 19 , 3 , 48 ] use cross-attention over spatio-temporal tokens and are a common choice for video [ 3 , 2 , 55 , 61 , 9 , 79 ] . Recent work [ 48 ] , improves attentive probe efficiency by removing redundant projections. Other approaches explicitly model time at the probe, enabling adaptation of image encoders to video [ 37 , 49 ] . DiST [ 49 ] adds a lightweight temporal encoder with shallow-to-deep feature fusion emphasizing high-level representations. In contrast, for dense prediction in images, DPT [ 51 ] fuses features from deep-to-shallow, emphasizing early-layer features. VDA extends this with a spatio-temporal head for inter-frame consistency. Despite this diversity, probe designs are typically developed and evaluated for specific tasks, with limited comparison across tasks or probe types. Moreover, probing and PEFT are studied separately, despite both operating on top of a frozen backbone. We instead evaluate representative probe designs jointly with PEFT methods and study how distributing the temporal context between backbone, probe and PEFT affects adaptation across appearance, motion and spatially dense video tasks. Parameter-Efficient Fine-Tuning. While probing adapts pretrained models through the prediction head, PEFT introduces learnable updates within a frozen backbone and is complementary to probe-based adaptation [ 19 , 48 ] . PEFT methods can be divided into four paradigms. Adapters [ 23 , 46 , 11 , 40 , 28 , 77 ] insert bottleneck modules within transformer blocks, with AdaptFormer [ 11 ] placing them in parallel with the MLP, rather than after it [ 23 ] . Additive methods [ 33 , 34 , 26 , 72 , 65 ] introduce learnable tokens into the input or intermediate representations, including frequency-aware (VFPT [ 72 ] ) and low-rank (BPT [ 65 ] ) variants. Selective [ 71 , 36 , 66 , 4 ] approaches instead update subsets of existing parameters, e.g. bias terms (BitFit [ 71 ] ), normalization (LayerNorm [ 4 ] ) or per-channel scale and shift (SSF [ 36 ] ). Reparameterization methods [ 24 , 75 , 27 , 38 , 60 ] inject low-rank residuals into existing weights. For example, LoRA [ 24 ] parameterizes the residual as the product of two low-rank matrices, while DoRA [ 38 ] further decomposes pretrained weights into magnitude and direction components, only updating the latter. While these methods are well-studied in image and language, their behavior for video remains less clear. Existing video-specific PEFT methods [ 37 , 44

Read the complete text