StreamMind : Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition

Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Donglin Bai, Zhibo Chen, Ting Cao · 2025

Read StreamMind : Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition by Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Donglin Bai, Zhibo Chen, Ting Cao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06220.txt.

Introduction

Talking with an AI about a live video feed demands answers at the speed of the stream itself. Xin Ding and seven co-authors built StreamMind in 2025, a video language model that watches at one hundred frames per second and speaks up on its own when something happens.

The bottleneck is that transformer costs grow much faster than the stream. Their answer is a gate: a small network watches the video and wakes the big language model only when a relevant event appears. A state-space feature extractor keeps the watching cost flat over time. On sports and first-person video tasks the system set new marks for both quality and speed, pointing toward always-on assistants for games and live media.

Catalog record

CREATORXin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Donglin Bai, Zhibo Chen, Ting Cao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06220

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

StreamMind : Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition Xin Ding, Hao Wu, Yifan Yang, Shiqi Jiang, Qianxi Zhang, Donglin Bai, Zhibo Chen, Ting Cao Abstract With the rise of real-world human-AI interaction applications, such as AI assistants, the need for Streaming Video Dialogue is critical. To address this need, we introduce StreamMind , a video LLM framework that achieves ultra-FPS streaming video processing (100 fps on a single A100) and enables proactive, always-on responses in real time, without explicit user intervention. To solve the key challenge of the contradiction between linear video streaming speed and quadratic transformer computation cost, we propose a novel perception-cognition interleaving paradigm named “event-gated LLM invocation” , in contrast to the existing per-time-step LLM invocation. By introducing a Cognition Gate network between the video encoder and the LLM, LLM is only invoked when relevant events occur. To realize the event feature extraction with constant cost, we propose Event-Preserving Feature Extractor (EPFE) based on state-space method, generating a single perception token for spatiotemporal features. These techniques enable the video LLM with full-FPS perception and real-time cognition response. Experiments on Ego4D and SoccerNet streaming tasks, as well as standard offline benchmarks, demonstrate state-of-the-art performance in both model capability and real-time efficiency, paving the way for ultra-high-FPS applications, such as Game AI and interactive media. The code and data is available at https://aka.ms/StreamMind . 1 Introduction The advancement of large foundation models is driving an increasing number of real-world human-AI interaction applications, such as AI home companions [ 41 , 3 ] and human-robot collaboration [ 54 , 63 ] . These applications heavily rely on streaming video understanding capabilities [ 71 , 44 , 14 , 61 , 26 ] , specifically streaming video dialogue [ 8 , 59 ] . Streaming video dialogue (StreamingVD) aims to continuously perceive incoming video streams and, based on user queries, provide proactive, real-time, and always-on responses without human intervention. For example, an AI companion could respond to user queries like “Guide me to fix this bicycle” or “Let’s watch and discuss the soccer game together”. Streaming video dialogue can be viewed as a generalization of standard offline video dialogue tasks [ 65 , 40 , 48 , 32 , 30 , 39 , 66 ] , extending video LLMs to broader real-world scenarios. The fundamental challenge of StreamingVD compared to its offline counterpart lies in the requirement for timing alignment between events and responses . That is, as the video progresses, the model is required to (1) proactively decide when to respond and (2) respond in real time before the next event occurs according to user queries, as illustrated in Fig. 2 . However, with current techniques, these two requirements are contradictory and cannot be achieved simultaneously. Figure 1 : The paradigm of existing per-step LLM invocation (left) versus our event-gated LLM invocation (right) for streaming video dialogue task. Our paradigm boosts frame rate up-to 100 fps on A100. ( R_{i} : response, f_{i} : frame, t: time) The seminal VideoLLM-Online [ 8 ] and VideoLLM-MoD [ 59 ] introduce a per-step LLM invocation paradigm for StreamingVD, shown in Fig. 1 . At each time step, it inputs all past frames and the user query to the LLM, to decide when to respond or remain silent. However, the O(n^{3}) 1 1 1 Invoking a quadratic-complexity LLM n times results in cubic complexity. computational complexity and limited context window are mismatched with the O(n) frame streaming, making real-time response difficult to achieve. More works [ 14 , 44 , 61 , 71 ] focus on enhancing the efficiency of offline video dialogue aiming to meet real-time requirements. However, similar to offline approaches, they depend on user queries to trigger responses rather than the model to do so proactively. Figure 2 : Comparison between streaming dialogue (bottom) and traditional dialogue tasks (top). Streaming dialogue proactively generates responses at appropriate time based on user queries and perception, whereas traditional dialogue requires human triggering for each response. To empower StreamingVD, this paper proposes a new perception-cognition interleaving paradigm, named event-gated LLM invocation , shown in Fig. 1 . It is inspired by the event perception mechanism of human brains [ 68 ] , which states that events are key components of human’s perception, attention, and memory 2 2 2 A formal definition of an event [ 68 ] : “A segment of time at a given location that is conceived by an observer to have a beginning and an end.” . We therefore propose an event-based selective cognition process. Rather than invoking the LLM at every time step, we introduce a Cognition Gate between the video encoder and LLM. The video encoder continuously perceives visual signals of each frame, and only when query-related events occur, the gate will open to invoke the LLM. Analogous to how a text prompt triggers an LLM, streaming visual signals lack a clear stopping point, unlike a finite text sequence. The cognition gate bridges this gap by acting as a mechanism to identify “physical world prompts” for the LLM. This new paradigm raises two major technical concerns, to achieve both high capability and efficiency. Firstly, existing video encoders in VideoLLMs (e.g., STC in VideoLLaMA) only capture local spatio-temporal features. This results in a linear increase in the number of feature tokens for streaming frames and subsequently a quadratic computation for the processing, failing to meet real-time requirements. Secondly, the Cognition Gate must decide to invoke the LLM or not, based on visual features and the user query. However, only considering efficiency and simply attending visual and query tokens (e.g., Cross Attention in Q-former [ 28 ] ) limits the gate capability to tasks like feature-matching [ 57 , 56 ] and retrieval [ 74 , 1 , 16 ] , struggle to make decisions with deeper semantic understanding. By addressing the challenges, we realize StreamMind , a novel framework for streamingVD, with the processing speed matching frame rate for the first time. It integrates two techniques. For the video encoder, inspired by the exceptional ability of the state-assisted module [ 22 ] in modeling continuous physical signals with constant cost, we propose the Event-Preserving Feature Extractor (EPFE) based on the state-assisted module. It generates a single perception token of event features to the gate, resulting in a constant perception cost. For the Cognition Gate, to leverage the world knowledge of LLM while meeting the real-time requirement, we propose a Shallow Layer Transfer method. The Gate is designed to reuse the shallow layers of the LLM and its autoregressive training to maximize the probabilities of “ <response/silence> ” tokens during training. To evaluate the performance of the StreamMind framework, we adopt evaluation metrics from VideoLLM-Online and a wide range of text generation evaluation metrics. Additionally, we design two novel metrics to comprehensively assess the model’s temporal responsiveness. Experiments on real-time Ego4D [ 21 ] and SoccerNet [ 20 ] demonstrate superior performance across all metrics. Furthermore, our model achieves state-of-the-art results on multiple offline benchmarks, including short- and long-term activity recognition and forecasting on the COIN [ 52 ] and Ego4D LTA [ 24 ] datasets. Notably, our framework breaks the ultra-FPS streaming video processing bottleneck, enabling

Read the complete text