T-Mask : Temporal Masking for Probing Foundation Models across Camera Views in Driver Monitoring

Thinesh Thiyakesan Ponbagavathi, Kunyu Peng, Alina Roitberg · 2025

Read T-Mask : Temporal Masking for Probing Foundation Models across Camera Views in Driver Monitoring by Thinesh Thiyakesan Ponbagavathi, Kunyu Peng, Alina Roitberg (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16207.txt.

Introduction

Thinesh Thiyakesan Ponbagavathi, Kunyu Peng, and Alina Roitberg tackle a practical weakness of driver monitoring systems: they fail when the camera moves. Deep learning foundation models generalize well when adapted lightly, by probing their final layers. But their robustness to unseen camera viewpoints was underexplored.

The authors adapt image foundation models to driver monitoring using a single training view, then evaluate directly on unseen perspectives with no further adaptation. They benchmark linear probes and advanced probing strategies for two foundation models against heavier fine-tuning approaches. Building on the results, they introduce T-Mask, an image-to-video probing method that masks temporal tokens to emphasize dynamic video regions. On a public driver activity dataset, T-Mask improved cross-view accuracy over strong probing baselines and over parameter-efficient fine-tuning, while adding no parameters. Gains were largest for underrepresented secondary activities.

Catalog record

CREATORThinesh Thiyakesan Ponbagavathi, Kunyu Peng, Alina Roitberg
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-16207

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

T-Mask : Temporal Masking for Probing Foundation Models across Camera Views in Driver Monitoring Thinesh Thiyakesan Ponbagavathi, Kunyu Peng, Alina Roitberg Abstract Changes of camera perspective are a common obstacle in driver monitoring. While deep learning and pretrained foundation models show strong potential for improved generalization via lightweight adaptation of the final layers (“probing”), their robustness to unseen viewpoints remains underexplored. We study this challenge by adapting image foundation models to driver monitoring using a single training view, and evaluating them directly on unseen perspectives without further adaptation. We benchmark simple linear probes, advanced probing strategies, and compare two foundation models (DINOv2 and CLIP) against parameter-efficient fine-tuning (PEFT) and full fine-tuning. Building on these insights, we introduce T-Mask – a new image-to-video probing method that leverages temporal token masking and emphasizes more dynamic video regions. Benchmarked on the public Drive&Act dataset, T-Mask improves cross-view top-1 accuracy by + 1.23 % +1.23\% over strong probing baselines and + 8.0 % +8.0\% over PEFT methods, without adding any parameters. It proves particularly effective for underrepresented secondary activities, boosting recognition by + 5.42 % +5.42\% under the trained view and + 1.36 % +1.36\% under cross-view settings. This work provides encouraging evidence that adapting foundation models with lightweight probing methods like T-Mask has strong potential in fine-grained driver observation, especially in cross-view and low-data settings. These results highlight the importance of temporal token selection when leveraging foundation models to build robust driver monitoring systems. Code and models will be made available here to support ongoing research. Accepted at ITSC 2025, ©IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. I Introduction Changes in camera perspective present a fundamental challenge for driver monitoring systems. Past studies have shown that the accuracy of a fully fine-tuned Convolutional Neural Network (CNN) can drop from [63%] to near-random performance when the camera angle facing the driver shifts [ 1 ] . Yet most driver activity recognition frameworks are still trained and evaluated under identical viewpoint conditions [ 2 , 3 , 4 , 5 ] . Where cross-view settings are considered [ 1 , 2 ] , they are often treated only as minimal side experiments and consistently reveal drastic performance drops. Fig. 1: High-level overview of our approach for secondary driver activity recognition under unseen viewpoints. We ask how well visual foundation models (FMs), fine-tuned with minimal parameters for in-cabin recognition, adapt to shifts in perspective. To address their limitations, we introduce T-Mask , a lightweight adaptation module that highlights dynamic, action-relevant tokens, improving robustness to viewpoint changes. Fig. 2: Overview of our proposed T-Mask framework for cross-view driver activity recognition. Left: Multi-view camera setup in the Drive&Act dataset showing diverse camera viewpoints. Middle: T-MASK identifies and masks static, low-motion tokens by computing consecutive embedding differences across frames, enabling the probe to focus on dynamic, action-relevant features. Right: The masking threshold is determined via distribution-guided selection using the mode of the embedding difference distribution plus a small offset \delta , ensuring a principled, data-efficient approach to separating static and dynamic tokens. Given the limitations of general driver activity models in dealing with viewpoint variance, foundation models [ 6 , 7 , 8 ] , which are pre-trained on extensive and diverse datasets in a self-supervised manner, offer a promising alternative for capturing more generalized representations. These models can be fine-tuned for various downstream tasks, setting new standards for state-of-the-art performance. While these models have been successful in general human activity recognition [ 9 , 10 , 11 , 12 , 13 , 14 ] , their ability to handle variations of camera perspective, especially in driving settings, remains unexplored. Given the great success of pretrained foundation models, we ask: can they also represent fine-grained driver activities and cope with changes in camera perspective? To investigate this, we systematically study cross-view generalization by probing two popular image foundation models (DINOv2 and CLIP) on one in-cabin camera view and evaluating them directly on unseen views. We focus primarily on probing approaches, as they offer the most extreme form of parameter efficiency by keeping the backbone entirely frozen and only adding a minimal number of new parameters, and consider different mechanisms, ranging from simple linear heads to more advanced attention-based probing methods. For completeness, we also benchmark against parameter-heavy PEFT [ 11 , 12 , 13 , 14 ] methods, which update small inserted modules or prompts inside the backbone, as well as full model fine-tuning [ 15 , 3 , 16 ] . While PEFT aims to reduce the number of trainable parameters, it still modifies a non-trivial portion of the network, making it substantially heavier than pure probing. We also extensively analyze cross-view performance relative to viewpoint shifts, quantifying pose differences with diverse metrics, such as geodesic [ 17 ] and SE(3) distances [ 18 ] . Building on this setup, we also introduce T-Mask , a new probing framework leveraging simple temporal masking aimed at suppressing static spatial features across frames, encouraging probes to focus on motion and dynamic cues – signals that are far more likely to generalize across different viewpoints. Our experiments conducted on the public large-scale Drive&Act dataset confirm that lightweight probing approaches, especially when augmented with T-Mask , are more robust to changes in camera perspective and outperform both PEFT methods and fully fine-tuned models. T-Mask establishes a new state-of-the-art on Drive&Act under cross-view and same-view regimes. The contributions of this work are summarized as follows: • We propose T-Mask , a temporal token masking strategy based on inter-frame token distances to suppress static tokens. This lightweight, plug-and-play method improves video classification performance when used with attention-based probing, without adding parameters. • We analyze cross-view generalization in driver activity recognition using the Drive&Act dataset, which features multi-view in-cabin recordings. Using the SE(3) geodesic distance between camera poses, we quantify viewpoint differences and show that recognition performance degrades with increased pose variation - highlighting the need for view-robust adaptation strategies. • We benchmark our method against both probing and parameter-efficient fine-tuning (PEFT) baselines. Our token masking strategy improves cross-view mean accuracy by +1-2 % over probing methods, and outperforms PEFT methods by up to +8% in mean accuracy, demonstrating superior generalization to viewpoint changes without additional parameters. II Related Work II-A Cross-view Driver Activity Recognition Cross-view driver activity recognition is critical for robust in-cabin understanding in autonomous vehicles, where camera perspectives vary due to sensor placement. While general activity datasets like NTU120 [ 19 ] , and Toyota Smart Home [ 20 ] support cross-view evaluation, Drive&Act [ 1 ] serves as a dedicated benchmark for fine-grained, multi-view driver activity recognition. Several methods have been proposed for general driver activity recognition using Convolutional neural networks [ 21 , 22 , 5 ] , multimodal fusion [ 23 , 3 ] , and attention-based methods [ 2 ] in the fixed view setting. While these approaches achieve strong performance in single-view scenarios, they do

Read the complete text