The Temporal Trap: Entanglement in Pre-Trained Visual Representations for Visuomotor Policy Learning

Nikolaos Tsagkas, Andreas Sochopoulos Duolikun Danier, Chris Xiaoxuan Lu, Oisin Mac Aodha · 2025

Read The Temporal Trap: Entanglement in Pre-Trained Visual Representations for Visuomotor Policy Learning by Nikolaos Tsagkas, Andreas Sochopoulos Duolikun Danier, Chris Xiaoxuan Lu, Oisin Mac Aodha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03270.txt.

Introduction

Nikolaos Tsagkas and co-authors name a flaw in using ready trained vision models for robot control in 2025. Such models read still images well and ignore time. Robot tasks unfold in sequence, so timing cues matter. The team calls the mismatch temporal entanglement. Features for a scene may not carry marks of task progress.

Policy success tracked how well hidden features coded progress steps. Richer time features alone did not fix control. A simple step that untangles time cues helped more. The authors urge builders to treat time as its own design axis. Robot eyes trained on photos need added time sense before they can steer hands well.

Catalog record

CREATORNikolaos Tsagkas, Andreas Sochopoulos Duolikun Danier, Chris Xiaoxuan Lu, Oisin Mac Aodha
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03270

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Temporal Trap: Entanglement in Pre-Trained Visual Representations for Visuomotor Policy Learning Nikolaos Tsagkas, Andreas Sochopoulos Duolikun Danier, Chris Xiaoxuan Lu, Oisin Mac Aodha Abstract The integration of pre-trained visual representations (PVRs) has significantly advanced visuomotor policy learning. However, effectively leveraging these models remains a challenge. We identify temporal entanglement as a critical, inherent issue when using these time-invariant models in sequential decision-making tasks. This entanglement arises because PVRs, optimised for static image understanding, struggle to represent the temporal dependencies crucial for visuomotor control. In this work, we quantify the impact of temporal entanglement, demonstrating a strong correlation between a policy’s success rate and the ability of its latent space to capture task-progression cues. Based on these insights, we propose a simple, yet effective disentanglement baseline designed to mitigate temporal entanglement. Our empirical results show that traditional methods aimed at enriching features with temporal components are insufficient on their own, highlighting the necessity of explicitly addressing temporal disentanglement for robust visuomotor policy learning. I INTRODUCTION The integration of pre-trained visual representations (PVRs) into visuomotor robot learning has emerged as a promising alternative to training visual encoders from scratch. Despite the promising results of these models in downstream robotic applications, including affordance-based manipulation [ 21 ] , semantically precise tasks [ 41 ] , and language-guided approaches [ 40 , 34 ] , their deployment in policy learning is still nascent. Overall, prior works have identified multiple issues revolving around the deployment of PVRs in robot learning. First, a consensus exists that we have not identified a single PVR that consistently leads to the best performance. This issue concerns not just the task at hand [ 28 ] but also the policy training paradigm [ 17 ] , with performance varying greatly. Similarly, policy robustness might suffer when the scene undergoes visual perturbations, even though the key motivation behind using PVRs is their generalisation capabilities [ 46 , 5 , 42 ] . We add to this list of issues that hinder policy performance by identifying the problem of temporal entanglement. In this work, we investigate a paradoxical phenomenon: models celebrated for their robustness on traditional computer vision benchmarks (e.g., classification) tend to underperform in robot learning tasks precisely because of this robustness. We show that the representations learned by such models exhibit temporal entanglement, not only between adjacent timesteps, but also over long-range temporal dependencies. Both entanglements correlate strongly with a model’s inability to predict task progression, a metric we find to be a strong predictor of policy success. Building on this insight, we introduce a simple yet powerful baseline for evaluating temporal disentanglement. Using this baseline, we demonstrate that conventional approaches, including feature augmentation, architectural changes to the policy, and alternative pre-training objectives, consistently fall short in addressing the temporal entanglement challenge. In summary, we make the following contributions: 1. Identifying temporal entanglement . We show that PVRs fall short in encoding temporal cues, even ones trained with temporal objective functions, which has a detrimental effect in visuomotor policy learning. We further demonstrate that these latent spaces lack a reliable notion of task progression. 2. Providing tools to quantify entanglement . We study both short and long-range temporal entanglement and reveal strong correlations with policy success. To this end, we propose a probing method that captures both types of entanglement and serves as a strong predictor of downstream performance. 3. Proposing a disentanglement baseline . We identify task-progression perception as a critical missing factor in current PVR-latent spaces and propose a simple yet effective approach to augment them with such a signal, yielding a significant boost in policy performance. Fig. 1 : PCA of features from an expert demonstration in Bin Picking across PVRs (Top row: ViT models; Bottom row: ResNet models). Frame colours align with trajectory stages, suggesting feature entanglement during the gripper descent and ascent , and during the gripper stop phase. Next to each PVR name we provide the success rate of the corresponding policy for the given task. II RELATED WORK In PVR-based visuomotor policy learning, the incorporation of temporal information remains underexplored. Augmentation with temporal perception can happen either at feature level or during training time. Feature Augmentation . While early fusion methods, such as stacking multiple frames before encoding [ 19 ] , are common in training visual encoders from scratch, late fusion ( i.e., processing frames individually and stacking their representations [ 35 ] ) has shown superior performance with fewer encoder parameters. Recent work [ 33 ] highlights that naive feature concatenation in latent space is insufficient; instead, approaches like FLARE [ 33 ] incorporate sequential embeddings and their differences, inspired by optical flow techniques. Nevertheless, concatenating sequential embeddings as input to policy networks has become standard in visuomotor policy learning [ 28 ] and SoTA generative policies [ 9 , 36 ] . However, a gap remains in leveraging PVR features, which are primarily designed for vision tasks, within this temporal framework. Loss Function Augmentation . A major limitation of many PVRs, in the context of visuomotor policy learning, is their inherit lack of temporal perception, as most are pre-trained on static 2D image datasets. Temporal perception can be added by employing loss functions that enforce temporal consistency during training ( e.g., R3M [ 26 ] and VIP [ 22 ] ), when training with video data. However, there is no clear consensus on the superiority of this approach compared to alternatives like masked-image modelling (MIM) ( e.g., MVP [ 45 , 17 ] and VC-1 [ 23 ] ). This disparity suggests that existing temporal modelling strategies may be insufficient in isolation. In later experiments, we evaluate PVRs trained with temporal information and demonstrate that methods trained with a time-agnostic paradigm achieve comparable performance. We hypothesise that this limitation arises from a lack of task-progression perception, which we address by incorporating positional encoding, a fundamental mechanism in many machine learning approaches. This straightforward operation has been instrumental in the success of Transformers [ 43 ] , implicit spatial representations [ 25 ] , and diffusion processes [ 16 ] . Our novelty lies in that we do not utilise this tool for encoding the position of an observation in the input stream, but rather in the global temporal stream of the rollout. Fig. 2 : Standard PVR-based behaviour cloning architecture, modified with out task-progression signal. We propose our disentangling baseline, which incorporates a task-progression signal, effectively disentangling features before the policy head. TABLE I : Overview of utilised PVRs. indicates models specifically trained for robotic tasks, while indicates a temporal training component. Dataset sizes are given in number of images, and † denotes number of frames from videos and ∗ denotes number of videos. Abbreviations: IN : ImageNet [ 31 ] , LVD : LVD-142M [ 27 ] , K : Kinetics [ 20 ] , E4D : Ego4D [ 12 ] , E4D+MNI [ 23 ] . Arch. PVR Training Objective Dataset (Size) ViT-B MAE [ 14 ] MIM IN (1.2M) VC-1 [ 23 ] MIM E4D+MNI (5.6M † ) DINOv1 [ 7 ] Self-Distillation IN (1.2M) iBOT [ 50 ] MIM+Self-Distillation IN (14M) DINOv2 [ 27 ] MIM+Self-Distillation LVD (142M) ViT [ 10 ] Supervised IN (14M)

Read the complete text