Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception
Dingcheng Huang, Xiaotong Zhang, Kamal Youcef-Toumi · 2026
Read Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception by Dingcheng Huang, Xiaotong Zhang, Kamal Youcef-Toumi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13176.txt.
Introduction
Dingcheng Huang and co-authors schedule perception modules by relevance for multimodal streaming in human-robot teamwork. Running every module on every frame piles up latency and hurts live performance. Robot events are sparse in information, so many modules waste work most frames. Their light scheduler uses past outputs to guess which modules the current scene needs. Tests cut latency against parallel pipelines and raised pose activation recall. Keyframe accuracy stayed high across the runs.
Catalog record
| CREATOR | Dingcheng Huang, Xiaotong Zhang, Kamal Youcef-Toumi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39130 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13176
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Perceive What Matters: Relevance-Driven Scheduling for Multimodal Streaming Perception Dingcheng Huang, Xiaotong Zhang, Kamal Youcef-Toumi Abstract In modern human-robot collaboration (HRC) applications, multiple perception modules jointly extract visual, auditory, and contextual cues to achieve comprehensive scene understanding, enabling the robot to provide appropriate assistance to human agents intelligently. While executing multiple perception modules on a frame-by-frame basis enhances perception quality in offline settings, it inevitably accumulates latency, leading to a substantial decline in system performance in streaming perception scenarios. Recent work in scene understanding, termed Relevance, has established a solid foundation for developing efficient methodologies in HRC. However, modern perception pipelines still face challenges related to information redundancy and suboptimal allocation of computational resources. Drawing inspiration from the Relevance concept and the information sparsity in HRC events, we propose a novel lightweight perception scheduling framework that efficiently leverages output from previous frames to estimate and schedule necessary perception modules in real-time based on scene context. The experimental results demonstrate that the proposed perception scheduling framework effectively reduces computational latency by up to 27.52% compared to conventional parallel perception pipelines, while also achieving a 72.73% improvement in MMPose activation recall. Additionally, the framework demonstrates high keyframe accuracy, achieving rates of up to 98%. The results validate the framework’s capability to enhance real-time perception efficiency without significantly compromising accuracy. The framework shows potential as a scalable and systematic solution for multimodal streaming perception systems in HRC. I Introduction Perception is a critical component enabling modern intelligent robots to achieve accurate scene understanding. In human-robot collaboration (HRC) scenarios, robotic agents must precisely perceive both the dynamic environment and human partners to provide appropriate assistance. HRC necessitates continuous execution and coordination of multiple perception modules and foundational models, enabling robots to interpret and respond to complex, dynamic environments intelligently [ 7 , 10 ] . Although substantial research has investigated efficient perception under latency and computational constraints, several open challenges persist. Modern perception pipelines typically execute modules frame by frame to optimize perception quality. While activating all available modules every frame ensures accurate offline perception, the computational costs introduce significant latency in streaming settings. In typical parallel pipelines, modules such as object detection and pose estimation execute concurrently, each producing outputs at its own rate. Heavy modules often run at lower frequency, displaying their most recent outputs until new ones become available [ 13 ] . However, modules are still invoked periodically based on readiness, regardless of whether their outputs contribute new or relevant information. For heavy modules, readiness-driven activation may lead to suboptimal timing, potentially missing critical scene information due to a lack of information-aware scheduling. Another research line leverages keyframe-based approaches, widely applied in video summarization, and key event detection [ 5 , 17 ] . Keyframes identified as important serve as triggers for activating perception modules, concentrating computational resources on critical moments while skipping redundant frames. However, keyframe-based approaches present limitations for real-time streaming perception. First, keyframe selection often relies on access to the full video sequence, making it unsuitable for real-time systems where future frames are unavailable. Second, common keyframe criteria—scene change, motion magnitude, visual saliency—are task-agnostic and often misaligned with the perception system’s informational needs, selecting irrelevant frames and causing suboptimal resource use without meaningful gains in situational awareness or decision quality. Research in efficient perception also aims to reduce cost by directly optimizing perception pipelines, primarily minimizing inference cost at the frame level through adjustment of configuration parameters. Common strategies involve fine-tuning settings, reusing intermediate features, or selectively processing input based on perceived importance. However, these efforts predominantly address cost reduction within individual frames without considering perception module scheduling to minimize overall system load. The decision of whether a module should be activated remains largely unexamined. For instance, when processing less informative frames, reducing model size or input resolution may lower inference time, yet bypassing non-essential frames entirely would more effectively eliminate unnecessary computations. Existing methods thus overlook the higher-level optimization of determining activation necessity, which remains crucial for comprehensive efficiency in multi-modal perception systems. Inspired by the relevance modeling framework, which emulates the human reticular activating system (RAS) to prioritize perceptual inputs based on task context and human intent [ 22 , 23 , 24 ] , and motivated by the information sparsity in HRC, we introduce a perception scheduling framework that selectively activates perception modules at each frame. The decision process leverages outputs from previous frames and scene dynamics to assess each module’s necessity in real time. When a module is not activated, a lightweight estimation, such as state prediction using motion models or filtering, approximates its output, maintaining perception stream continuity without full computational cost. A reward function governs the scheduling policy by quantifying the trade-off between expected information gain and computational efficiency, enabling resource-aware and context-sensitive perception under real-time constraints. The proposed framework treats all schedulable perception modules as elements within a shared resource pool referred to as the Perception Toolkit. Each module is associated with a reward function quantifying its utility for the current frame. Although exact reward formulations may vary across modules, all follow the same principle: information gain is estimated using a module-specific function, followed by a penalty reflecting computational cost. Once individual rewards are computed, the scheduler evaluates all possible activation combinations and selects the subset maximizing cumulative reward. To demonstrate effectiveness, the current work focuses on two representative modules: object detection and full-body human pose estimation. However, the scheduling principle’s generality allows extension to other perception modules and foundation models, such as vision-language models (VLMs), provided appropriate reward formulations are defined. Most importantly, the framework shows potential to mitigate concurrent contention among modules, accommodating larger module sets within fixed computational budgets. In summary, this paper presents four key contributions toward efficient and adaptive perception in HRC: (1) We introduce the novel concept of perception scheduling within multi-modal perception systems, addressing the fundamental challenge of optimizing module activation to minimize overall computational load. (2) We propose a novel methodology for estimating module activation utility based on information theory, offering a versatile principle applicable to a wide range of perception modules. (3) We develop a novel evaluation metric for real-time perception scenarios, accounting for latency effects and providing a practical performance benchmark. (4) We validate the proposed framework through comprehensive experiments across various streaming video domains, demonstrating improvements in computational efficiency and inference latency while maintaining perceptual quality for collaborative tasks. II Related Works We identified three main fields of work that shared the same goal of optimizing perception systems: keyframe-based methods, computation-aware efficient perception, adaptive sampling and sensor scheduling. To the best of our knowledge, our work is the first to introduce a perception scheduling framework for HRC perception tasks. II-A Keyframe Detection Keyframe-based methods are widely adopted in video summarization, segmentation, and activity recognition to reduce redundancy while preserving semantic content. Early techniques rely on motion cues [ 4 , 11 ] , statistical histogram analysis [ 3 ] , or feature-space clustering [ 26 ] to select visually distinct or representative frames. Other methods employ optimization-based criteria such as sparse reconstruction [ 15 ] to ensure coverage with minimal redundancy. More recent approaches incorporate deep learning, including the unsupervised learning-based approach to detect video highlights [ 20 ]