LiveStar: Live Streaming Assistant for Real-World Online Video Understanding

Zhenyu Yang, Kairui Zhang, Yuhang Hu, Bing Wang, Shengsheng Qian, Bin Wen, Fan Yang, Tingting Gao, Weiming Dong, Changsheng Xu · 2025

Read LiveStar: Live Streaming Assistant for Real-World Online Video Understanding by Zhenyu Yang, Kairui Zhang, Yuhang Hu, Bing Wang, Shengsheng Qian, Bin Wen, Fan Yang, Tingting Gao, Weiming Dong, Changsheng Xu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05299.txt.

Introduction

Zhenyu Yang and co-authors present LiveStar, a live streaming assistant that watches video as it flows and replies at the right moments. Offline video models can scan a whole clip first, but a live aid must act frame by frame and choose when to speak.

LiveStar uses adaptive streaming decoding to stay always on and proactive. A training plan aligns video and language step by step, so replies stay in sync with events. The design targets live help where late or off-beat replies fail, such as guided tasks and live event support.

Catalog record

CREATORZhenyu Yang, Kairui Zhang, Yuhang Hu, Bing Wang, Shengsheng Qian, Bin Wen, Fan Yang, Tingting Gao, Weiming Dong, Changsheng Xu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE103 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-05299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LiveStar: Live Streaming Assistant for Real-World Online Video Understanding Zhenyu Yang, Kairui Zhang, Yuhang Hu, Bing Wang, Shengsheng Qian, Bin Wen, Fan Yang, Tingting Gao, Weiming Dong, Changsheng Xu Abstract Despite significant progress in Video Large Language Models (Video-LLMs) for offline video understanding, existing online Video-LLMs typically struggle to simultaneously process continuous frame-by-frame inputs and determine optimal response timing, often compromising real-time responsiveness and narrative coherence. To address these limitations, we introduce LiveStar, a pioneering live streaming assistant that achieves always-on proactive responses through adaptive streaming decoding. Specifically, LiveStar incorporates: (1) a training strategy enabling incremental video-language alignment for variable-length video streams, preserving temporal consistency across dynamically evolving frame sequences; (2) a response-silence decoding framework that determines optimal proactive response timing via a single forward pass verification; (3) memory-aware acceleration via peak-end memory compression for online inference on 10+ minute videos, combined with streaming key-value cache to achieve 1.53× faster inference. We also construct an OmniStar dataset, a comprehensive dataset for training and benchmarking that encompasses 15 diverse real-world scenarios and 5 evaluation tasks for online video understanding. Extensive experiments across three benchmarks demonstrate LiveStar’s state-of-the-art performance, achieving an average 19.5% improvement in semantic correctness with 18.1% reduced timing difference compared to existing online Video-LLMs, while improving FPS by 12.0% across all five OmniStar tasks. Our model and dataset can be accessed at https://github.com/yzy-bupt/LiveStar . 1 Introduction Recent advancements in Large Vision-Language Models (LVLMs) [ 7 , 49 , 68 , 73 , 13 ] have substantially propelled the development of Video Large Language Models (Video-LLMs) [ 2 , 35 , 25 , 63 , 52 ] . These cutting-edge models have been instrumental in pushing the boundaries of multimodal AI systems through innovations in spatial-temporal reasoning [ 8 , 34 , 43 ] , memory-enhanced processing [ 72 , 46 , 17 ] , and long-term video understanding [ 51 , 62 , 74 ] , particularly enhancing capabilities in offline scenarios. Building on these advances, researchers are increasingly exploring live streaming assistants for online video understanding [ 5 , 55 , 28 , 9 , 54 , 40 ] , which remain always-on, reason through time, and provide real-time responses. However, existing online Video-LLMs [ 40 , 77 , 57 ] struggle to process frame-by-frame inputs and determine response timing, as shown in Fig 1 (a). To effectively process continuous video streams and generate timely responses, such models must possess temporal decision-making capabilities—responding at contextually appropriate moments rather than producing repetitive outputs or default "I don’t know" responses for each frame. To address this, VideoLLM-online [ 5 ] designs a streaming EOS (End-Of-Sequence) prediction mechanism (Fig. 1 (b)). It ingests video streams, determines response timing based on user queries, and generates EOS tokens to mark silence intervals. Subsequent advancements like VideoLLM-MoD [ 55 ] and LION-FS [ 28 ] have further refined this EOS-based framework, achieving improved computational efficiency and response accuracy. Figure 1: Illustration of online video understanding. (a) Taking the RNG task as an example, online video understanding requires Video-LLMs to handle continuous streams and output at appropriate times; (b) Existing methods overly rely on learning the EOS token, leading to poor inference performance; (c)-(e) LiveStar establishes an effective response-silence training and inference framework by SCAM and SVeD without compromising basic video understanding capabilities. However, current online Video-LLMs exhibit excessive reliance on the EOS token, which leads to four critical issues: (1) Response-Silence Imbalance : Frames requiring EOS tokens as output vastly outnumber those generating normal responses [ 9 ] . For example, in a 1-minute video at 3 FPS with 5 response intervals, the response-to-silence ratio becomes 1:35. (2) Consecutive Frame Inconsistency : As shown in Fig. 1 (b), adjacent visually similar frames may produce conflicting outputs—one generating a full narration while the other outputs only the EOS token. This inconsistency disrupts model convergence during fine-tuning. (3) Pre-training Misalignment : While pretraining aligns image-text pairs, silent states enforce a mapping from input frames to EOS tokens, contradicting the pretraining objective of meaningful visual-language correspondence. (4) Vocabulary Confusion : Since the EOS token is embedded in the vocabulary as a regular token, its frequent appearance in responses pollutes semantic coherence, introducing ambiguity and conflicting with normal outputs. These issues hinder model training effectiveness and degrade base video understanding capabilities, as the forced response-silence mechanism compromises the visual-language alignment. Motivated by this critical gap, we aim to address Challenge 1 : How to establish an effective response-silence training and inference framework without compromising basic video understanding capabilities? Moreover, online Video-LLMs face challenges due to limited diversity in training data and evaluation scope, failing to fully encompass the online applications encountered in real-world scenarios. For instance, prominent models [ 5 , 55 , 28 ] primarily focus on first-person video analysis through training on Ego4D [ 14 ] . Although StreamMind [ 9 ] extends its reach to sports scenarios with SoccerNet [ 12 ] , its scope remains severely constrained. Recent efforts like SVBench [ 65 ] , OVO-Bench [ 29 ] , and StreamBench [ 58 ] have pioneered online evaluation by assessing models’ ability to process videos synchronously during playback. However, these benchmarks focus exclusively on video question answering and lack coverage of diverse real-world tasks. In practice, beyond first-person devices, online scenarios span live streaming platforms, surveillance systems, and cinematic tools, requiring capabilities like real-time interactive editing [ 67 , 66 ] , streaming narration, and temporal grounding in continuous streams. Current evaluation benchmarks present three critical limitations: (1) Narrow scenario coverage limited to specific domains, (2) Single-task evaluation focused on video question answering, and (3) Offline assessment of temporal reasoning and response timing. These shortcomings motivate us to tackle Challenge 2 : How to construct a comprehensive dataset and benchmark encompassing diverse real-world scenarios and tasks for holistic evaluation of online video understanding? To deal with the above challenges, we introduce LiveStar , an innovative live streaming assistant that enables online video understanding across diverse scenarios, delivering context-aware responses at semantically relevant moments through adaptive streaming decoding. For Challenge 1 , we establish a streaming response-silence paradigm featuring two key innovations: (1) a streaming video-language alignment strategy using Streaming Causal Attention Masks ( SCAM ) during training, and (2) a Streaming Verification Decoding ( SVeD ) mechanism for inference. During training, we construct interleaved frame-caption sequences where revealed video frames progressively align with their corresponding captions through causal masked attention constraints (Fig. 1 (d)), maintaining consistency with standard multimodal pre-training paradigms. For streaming inference, SVeD employs single-pass verification to determine optimal response timing: it suppresses redundant outputs by comparing current content with previous responses, maintaining narrative continuity through strategic silence. To handle extended video contexts, we implement a peak-end memory compression strategy that prioritizes salient keyframes and final captions, enabling efficient processing of long videos by focusing on semantically rich content while accelerating inference through streaming key-value cache. For Challenge 2 , we introduce OmniStar , a dataset for training and benchmarking that encompasses diverse real-world scenarios and evaluation tasks in online video understanding. The dataset includes carefully annotated real-time narration, streaming QA, and streaming video grounding tasks on 20,137 videos, evaluated through an online setting that better simulates real-world conditions compared to

Read the complete text