Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos
Yanlai Yang, Mengye Ren · 2025
Read Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos by Yanlai Yang, Mengye Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-12254.txt.
Introduction
Yanlai Yang and Mengye Ren wrote a machine learning paper in two thousand twenty-five on learning from first-person video streams. Self-supervised learning promises good visual features from raw, unedited real-world footage. Most past work used still images or fake streams, which are far simpler than a day seen through someone's eyes.
Their method, Memory Storyboard, borrows from how human memory cuts experience into events. Recent frames are grouped into segments, held in a short-term store, then passed to a long-term store for replay. On two real first-person data sets, learning from storyboard frames gave richer features than rival methods.
Catalog record
| CREATOR | Yanlai Yang, Mengye Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-12254
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Memory Storyboard: Leveraging Temporal Segmentation for Streaming Self-Supervised Learning from Egocentric Videos Yanlai Yang, Mengye Ren Abstract Self-supervised learning holds the promise to learn good representations from real-world continuous uncurated data streams. However, most existing works in visual self-supervised learning focus on static images or artificial data streams. Towards exploring a more realistic learning substrate, we investigate streaming self-supervised learning from long-form real-world egocentric video streams. Inspired by the event segmentation mechanism in human perception and memory, we propose “Memory Storyboard” that groups recent past frames into temporal segments for more effective summarization of the past visual streams for memory replay. To accommodate efficient temporal segmentation, we propose a two-tier memory hierarchy: the recent past is stored in a short-term memory, and the storyboard temporal segments are then transferred to a long-term memory. Experiments on real-world egocentric video datasets including SAYCam [ 55 ] and KrishnaCam [ 52 ] show that contrastive learning objectives on top of storyboard frames result in semantically meaningful representations which outperform those produced by state-of-the-art unsupervised continual learning methods. Figure 1: Our proposed Memory Storyboard framework for streaming SSL from egocentric videos. Similar frames are being clustered into temporal segments and their labels (text information for illustration purpose only) are updated in the long-term memory buffer for replay. SSL involves contrastive learning at both the frame and temporal segment levels. 1 Introduction Humans are capable of learning continuously from a stream of unlabeled and uncurated perceptual inputs, such as video data, without needing to iterate through multiple exposures or epochs. Since early infancy, humans have accumulated knowledge about the world through a continuous flow of raw visual observations. This capability contrasts sharply with the training paradigm of current methods in self-supervised learning (SSL). While recent SSL approaches have made great strides in learning from large unlabeled datasets [ 11 , 19 , 12 , 9 , 7 , 25 , 3 , 26 ] , they still predominantly rely on static and curated image datasets, such as ImageNet [ 13 ] , and needs multiple epochs of training for effective learning. This difference in paradigm raises a compelling question: how can we learn good visual representations in a streaming setting—learning from visual inputs in their original temporal order without cycling back? Motivated by the differences of mechanisms between human learning and standard SSL, we aim to build learning algorithms that can efficiently learn visual representations and concepts from streaming video. One especially relevant mechanism in the human brain is event segmentation [ 37 , 70 ] , where we spontaneously segment visual streams into hierarchically structured events and identify the event boundaries. Take your recent vacation trip as an example—you probably remember separate events and activities like exploring a city, dining at a local restaurant, or relaxing at the beach. The event segmentation mechanism helps us organize memories, recall specific moments, and summarize from lengthened experiences [ 68 , 69 ] . Drawing inspiration from the way we organize our memory in the brain, we introduce Memory Storyboard , a novel approach for streaming self-supervised learning. Memory Storyboard features a temporal segmentation module, which groups video frames into semantically meaningful temporal segments, resembling the automatic event segmentation of human cognition. Through our temporal contrastive learning objective, these temporal segments effectively facilitate representation learning in streaming videos. To accommodate efficient temporal segmentation, we propose a two-tier hierarchical memory: temporal segmentation in the short-term memory are used to update the temporal class labels in the long-term memory, and a training batch consists of samples mixed from both memories. A high-level diagram of the algorithm is shown in Figure 1 . We conduct experiments on the SAYCam [ 55 ] and KrishnaCam [ 52 ] datasets of real-world egocentric videos. Memory Storyboard outperforms state-of-the-art unsupervised continual learning methods on downstream image classification and object detection tasks, and significantly reduces the gap between streaming learning and the less flexible IID learning that requires persistent storage of the entire prior video data. We also experiment with different buffer sizes and batch sizes and offer insights on the optimal training batch composition under different memory constraints. We summarize our contributions as follows: 1. We introduce Memory Storyboard, a novel streaming SSL framework that features temporal segmentation and a two-tier memory hierarchy for efficient learning and temporal abstraction. 2. We demonstrate that Memory Storyboard achieves state-of-the-art downstream performance when trained on real-world egocentric video datasets. Among all the streaming self-supervised learning methods we evaluated, Memory Storyboard is the only one that is competitive with or even outperforms IID training when trained on these datasets. 3. We study the effects of training factors including subsampling rate, average segment length, memory buffer size and training batch composition. These studies provide insight for more efficient streaming learning from videos. In particular, we explore the optimal composition ratio of the training batch from short-term vs. long-term memory, under different memory constraints. Larger batches from long-term memory improve performance when we can afford a large memory bank, while smaller batches can help prevent overfitting when we have a small memory bank. 2 Related Work Unsupervised Continual Learning. Unsupervised Continual Learning (UCL) [ 44 , 53 ] aims at learning a good representation through an unlabeled non-stationary data stream. Existing works in UCL, notably CaSSLe [ 17 ] and Osiris [ 73 ] , assume that the data stream is composed of a series of episodes, and a stationary data distribution within each episode. This is not as naturalistic and human-like as our streaming setting, where the data distribution changes continuously in through the data stream, and each image appear in the data stream only once. Meanwhile, we showed that existing UCL methods are also effective in our streaming video setting, and can be used together with the supervised contrastive objective. Self-Supervised Learning. A large number of self-supervised representation learning methods in computer vision follows the contrastive learning framework [ 39 , 36 , 56 , 26 , 11 , 12 ] which maximizes the agreement of representations of two augmented views of the same image and minimizes that of different images. Extending this idea, the supervised contrastive (SupCon) method [ 30 ] uses the labels as an extra supervision signal to get multiple positive crops for each anchor image. Other recent self-supervised learning works include pretext tasks [ 14 , 38 , 18 , 41 ] , feature space clustering [ 8 , 9 , 45 ] , distillation with asymmetric architectures [ 19 , 12 ] , redundancy reduction [ 71 , 7 ] , and masked autoencoding [ 25 ] . Most relevant of these to our work, Orhan et al., 2020 proposes the temporal classification objective, which outperforms contrastive learning objectives on the SAYCam dataset [ 55 ] . Our work enhances the temporal classification method with using a more flexible supervised contrastive objective, and leveraging temporal segmentation [ 42 , 1 ] , which have been used extensively in video summarization [ 74 , 72 , 48 ] . Streaming Learning from Videos. While a number of recent papers have studied streaming learning from images [ 22 , 24 , 23 , 6 ] , limited works have investigated the problem of streaming