PEARL: Personalized Streaming Video Understanding Model
Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, YiFan Zhang † \dagger, Yuheng Li, Wentao Zhang · 2026
Read PEARL: Personalized Streaming Video Understanding Model by Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, YiFan Zhang † \dagger, Yuheng Li, Wentao Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20422.txt.
Introduction
Zheng and co-authors define a new task: personalized streaming video understanding. People learn new faces and objects as a stream unfolds. Current personalization methods, though, rely on static images or offline video. Their PEARL-Bench tests responses to personal concepts at exact timestamps. It holds 132 videos and 2,173 fine-grained annotations. Modes cover single frames and actions across continuous frames. They also propose PEARL, a plug-and-play model for the task. The work targets real-time, interactive personalized AI assistants.
Catalog record
| CREATOR | Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, YiFan Zhang † \dagger, Yuheng Li, Wentao Zhang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61689 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20422
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PEARL: Personalized Streaming Video Understanding Model Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, YiFan Zhang † \dagger, Yuheng Li, Wentao Zhang Abstract Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current multimodal personalization methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model’s ability to respond to personalized concepts at exact timestamps under two modes: (1) Frame-level, focusing on a specific person or object in discrete frames, and (2) a novel Video-level, focusing on personalized actions unfolding across continuous frames. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. Concept diversity and annotation quality are strictly ensured through a combined pipeline of automated generation and human verification. To tackle this challenging new setting, we further propose PEARL, a plug-and-play, training-free strategy that serves as a strong baseline. Extensive evaluations across 8 offline and online models demonstrate that PEARL achieves state-of-the-art performance. Notably, it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be a highly effective and robust strategy. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants. Code is available at https://github.com/Yuanhong-Zheng/PEARL . † † footnotetext: * Equal contribution. \dagger Project leader. Corresponding authors. Figure 1 : Comparison between our proposed P ersonalized S treaming V ideo U nderstanding (PSVU) task and traditional Personalized Image/Video Understanding tasks. Unlike traditional settings, PSVU better aligns with real-world scenarios by featuring continuous streaming video inputs, dynamically defined concepts, and multi-turn conversations regarding these customized concepts. 1 Introduction Recent advancements in Vision-Language Models (VLMs) [ 35 , 7 , 19 , 45 , 15 , 47 ] have remarkably expanded the boundaries of multimodal understanding, empowering models to recognize and interact with personalized, user-specific concepts. Despite these strides, current personalization methods [ 4 , 24 , 16 , 32 , 37 , 5 , 25 , 39 , 18 ] remain fundamentally constrained. As shown in Fig. 1 , approaches such as Yo’LLaVA [ 24 ] and MC-LLaVA [ 4 ] are mainly designed for static image-text tasks. Furthermore, while PVChat [ 32 ] pioneers personalized video understanding, it operates strictly in offline settings and only supports single turn interaction, failing to accommodate the open-ended, streaming nature of real-world environments. In contrast, humans continuously recognize new individuals and objects, forming memories over time as they process the world as a seamless visual stream. This fundamental cognitive mechanism highlights a critical limitation of existing methods, which remain confined to static images or pre-recorded videos. Bridging this gap is not merely a technical step, but an essential prerequisite for the next generation of personalized AI assistants [ 2 ] : such systems must be capable of handling streaming visual inputs and delivering real-time, interactive, and personalized responses in dynamic real-world environments [ 14 ] . For instance, in customized fitness coaching (Fig. 1 ), an AI assistant must continuously monitor a user’s specific weightlifting actions across an video stream to provide instant, tailored form correction. This real-time, streaming personalization capability is indispensable for deploying truly practical AI assistants. To bridge this gap, we firstly propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark specifically designed to evaluate personalized streaming video understanding. Unlike traditional offline tasks, PEARL-Bench distinguishes itself through two core properties: (1) Continuous Temporal Precision , requiring models to localize and reason about personalized concepts at exact timestamps within an ongoing stream; and (2) Interactive Concept Definition , challenging models to grasp user-specific concepts dynamically defined on-the-fly, rather than relying on predefined pools. To thoroughly assess model capabilities, the benchmark evaluates two modes: (1) Frame-level Personalization, focusing on the continuous recognition and reasoning of a specific person or object appearing across discrete frames; and (2) a novel Video-level Personalization, which goes beyond static appearances to focus on specific, customized actions that unfold across continuous frames. PEARL-Bench comprises 132 unique videos, and 2,173 fine-grained annotations with precise timestamps. The video data is carefully curated through a combination of expert manual collection and rigorous programmatic synthesis pipelines. By sourcing data from diverse domains including anime, movies, reality shows, and digital humans, we ensure both extensive concept diversity and high annotation quality. The challenging PSVU task presents significant efficiency and architectural hurdles for existing models, as they struggle to maintain streaming visual context and instantly acquire new concepts without computationally expensive retraining. To tackle this, we propose PEARL, a training-free plug-and-play framework designed to serve as a strong baseline. Specifically, PEARL features a Dual-grained Memory System that explicitly decouples concept-centric knowledge from stream-centric observations, incrementally archiving continuous video clips while dynamically registering user-defined concepts. To ensure fast and accurate response, we further introduce a Concept-aware Retrieval Algorithm that leverages stored concept descriptions to precisely retrieve relevant historical visual evidence. Consequently, without any parameter updates, PEARL seamlessly empowers off-the-shelf VLMs to deliver real-time, personalized responses in continuous video streams. Extensive evaluations demonstrate that PEARL establishes a new state-of-the-art among 8 offline and online models. Notably, equipped with PEARL drives consistent improvements across 3 distinct architectures, yielding an average performance gain of 13.79% at the frame-level and 12.80% at the video-level, thereby proving its effectiveness and robustness. We summarize our contributions as follows: • New Task and Benchmark: We are the first to propose and formally define the novel task of Personalized Streaming Video Understanding. To facilitate evaluation in this direction, we introduce PEARL-Bench, the first comprehensive benchmark specifically designed for this challenging setting. • Novel Framework: We propose PEARL, a novel, training-free, and plug-and-play method. By seamlessly integrating into existing models, it demonstrates remarkable effectiveness and robustness across multiple architectures. • State-of-the-Art Performance: Extensive experiments show that PEARL achieves state-of-the-art results compared to 8 offline and online video understanding methods. We hope this work inspires the field of VLM personalization and paves the way for next-generation interactive AI assistants. 2 Related Works Personalized VLMs. As the capabilities of Vision-Language Models (VLMs) continue to advance [ 35 , 7 , 19 , 45 , 15 , 47 , 48 , 29 , 23 ] , growing attention has been increasingly directed toward unleashing their potential to serve as personalized AI assistants [ 17 , 10 , 36 , 28 , 20 ] . Existing VLM personalization efforts can be broadly categorized into three areas: personalized image understanding, combined with personalized generation and personalized video understanding. Previous research has predominantly focused on personalized image understanding. The paradigm can be summarized as finetune-based [ 3 , 24 , 4 , 39 ] , Retrieval-Augmented Generation (RAG-based [ 16 , 37 ]