Forget, Anticipate and Adapt: Test Time Training for Long Videos Corresponding Author.
Rajat Modi, Sebastian Noel, Xin Liang, Yogesh S. Rawat · 2026
Read Forget, Anticipate and Adapt: Test Time Training for Long Videos Corresponding Author. by Rajat Modi, Sebastian Noel, Xin Liang, Yogesh S. Rawat (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26515.txt.
Introduction
Rajat Modi, Sebastian Noel, Xin Liang, and Yogesh S. Rawat adapt video models while they run. Test-time training updates a model's weights on the incoming video itself, using a self-teaching task that needs no labels. For long videos, older methods update on a sliding window of past frames. The computing cost grows with the window and becomes unworkable for videos hours long.
Their Frame Forgetting Network uses only three frames at a time. It keeps the frame leaving the window, the current frame, and the next one. Temporal context survives despite the tiny window. A surprise measure, defined mathematically, gauges how much new information each frame carries. That measure drives an adaptive window that spends effort only when the scene changes. The team also built EpicTours, a data set of walking tours up to three hours long. Earlier sets held only minutes of video.
Catalog record
| CREATOR | Rajat Modi, Sebastian Noel, Xin Liang, Yogesh S. Rawat |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26515
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Forget, Anticipate and Adapt: Test Time Training for Long Videos Corresponding Author. Rajat Modi, Sebastian Noel, Xin Liang, Yogesh S. Rawat Abstract Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute. We present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a ‘surprise’ metric: how much ‘new information’ the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFN’s empirical effectiveness on dense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos. The project page can be found at https://github.com/rajatmodi62/ffn . 1 Introduction Typically, machine learning models only update their weights during training, but freeze them during inference [ 1 , 2 ] . However, as humans, we possess the ability to continuously learn and adapt to our ever-changing environment. Test Time Training (TTT) shows the promise of bringing a similar adaptive capability to artificial intelligence, allowing a model to continuously learn and update its weights even while testing , entirely without the need for ground-truth labels. However, applying TTT to video processing presents a unique set of challenges. Existing online distillation methods are based on a student-teacher setup [ 3 ] , with the teacher usually deployed on a remote server and the student on a local device. However, this is a bottleneck in offline real-world scenarios, such as disaster-prone regions , where server connectivity may be limited. Consequently, the key challenge lies in determining how to perform TTT on videos locally, on-device, and computationally cheaply. This is especially critical for dense computer vision tasks, such as segmentation, where each pixel matters, and should ideally be achieved utilizing a single model. Historically, there have been two approaches towards adaptation: either (i) train a dedicated video model specifically for the task, or (ii) start from a pre-trained image model and adapt it accordingly. Unfortunately, large-scale pre-training datasets for video remain limited , and efforts to repurpose image models are inherently constrained by the absence of explicit temporal information. Furthermore, existing TTT methods for videos rely on a sliding window approach [ 4 ] . A sliding window serves as a fixed-size temporal buffer that holds a defined number of preceding frames and moves forward step by step as the video advances. Because the model re-evaluates all frames currently inside this window, it performs a duplicate , highly expensive computation at every single timestep. This redundant computation is far too prohibitive for long videos, thereby limiting practical deployment To overcome these limitations, we propose the Frame Forgetting Network (FFN). Our approach is built on the core insight that as the sliding window progresses, only one new frame enters, and one frame exits. Thus, a model only needs to pay the computational cost of processing these crucial frames instead of recalculating the entire window for every time-step. Our FFN consists of two core components: a Memory Restoration Mechanism (MRM) and an Adaptive Window Algorithm (AWA). The MRM allows the model to actively ‘forget’ its adaptation on past frames exiting the window and ‘restore’ the model’s original predicted features using a temporal module. Next, the AWA algorithm can dynamically ‘anticipate’ when adaptation is required, improving the management of computational load. Through extensive experiments across 11 datasets, we validate FFN’s competitive performance on dense segmentation tasks, strong generalization on depth estimation, and ability to learn stable representations during recurrent video-processing. 2 Frame Forgetting Network First, we shall talk about our problem statement, cover preliminaries on TTT, and then discuss our Frame Forgetting Network (FFN). 2.1 Problem statement Given a streaming long-video \mathcal{V}=\{x_{1},x_{2},x_{t},...,x_{n}\} , a model can access only the past frames x_{1},x_{2},...,x_{t-1} , the current frame x_{t} , but not the future. The problem is focused on adaptation: for x_{t} , how does the model decide when to ‘adapt’ or not. This adaptation should help improve downstream performance and be cheaper to compute. In this work, we focus on adaptation using TTT, an instantiation that allows a model to even update its weights during inference. 2.2 Preliminaries TTT involves two phases, a) Training Time Training and b) Test Time Training. Training Time Training : Fig 1 (i) illustrates how a TTT setup works. We consider a model containing backbone f , a downstream head h and a self-supervised (SSL) head g specializing in some task that does not require external labels (e.g., image reconstruction). First, we train (f,g,h) jointly in a train set, which consists of densely-annotated images. This first phase is known as training-time-training [ 4 ] . Figure 1 : (i) Test Time Training setup: Here f is image-backbone, g is SSL head, h downstream head. Learning is self-supervised, where input test sample x_{t} is transformed into x_{t^{\prime}} and compared again with x_{t} . (ii) TTT on videos relies on sliding windows, we denote two such windows W_{t} , W_{t+1} . Notice that they contain a lot of overlapping frames, requiring N operations everytime. (iii) (Ours) We can perform TTT by gradually operating on only 3 frames via a ‘forget, anticipate, adapt’ procedure. Test Time Training : Here, the model is given access to a test-video consisting of several frames. The model looks at an incoming frame x_{t} and tries to reconstruct it. As in Fig 1 , this involves feed-forwarding x_{t} through the backbone f , then through the SSL-head g , thus computing the RGB reconstruction x^{\prime}_{t} . Next, we estimate the reconstruction loss \ell_{s}(x^{\prime}_{t},x_{t}) , where x^{\prime}_{t}=g{\circ}f(x_{t}) . This may be used to update f,g over several iterations. The downstream head h is kept frozen throughout. This is known as test-time-training . 2.3 The Principle of locality Figure 2 : The Principle of Locality : The first 3 frames (outdoors, marked in red) may not be directly relevant to last 3 frames (indoors, marked in green), therefore during TTT, our model neglects them. Best viewed in color. Adaptation in videos follows a ‘principle of locality’ [ 4 ] . Intuitively, a frame at t=1 (say indoors) may not be relevant to the frame at t=5k (say outdoors). Consider a current frame x_{t} . Instead of looking at the entire past timesteps , existing TTT methods use a sliding window of size k , which we denote by