StreamSense: Streaming Social Task Detection with Selective Vision–Language Model Routing Conference: Proceedings of the ACM Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates Proceedings of the ACM Web Conference 2026 (WWW ’26), April 13–17, 2026, Dubai, United Arab Emirates DOI: 1
Han Wang, Deyi Ji, Lanyun Zhu, Jiebo Luo, Roy Ka-Wei Lee · 2026
Read StreamSense: Streaming Social Task Detection with Selective Vision–Language Model Routing Conference: Proceedings of the ACM Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates Proceedings of the ACM Web Conference 2026 (WWW ’26), April 13–17, 2026, Dubai, United Arab Emirates DOI: 1 by Han Wang, Deyi Ji, Lanyun Zhu, Jiebo Luo, Roy Ka-Wei Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22738.txt.
Introduction
Han Wang and four co-authors built StreamSense, a live monitor that reads social cues in streams and calls in a big model only when needed. Live platforms must react to social signals in real time, from partial video, text, and audio that arrive out of sync. Their 2026 preprint pairs a light streaming encoder with selective routing.
Hard cases go to a vision-language expert; easy ones stay cheap. That split keeps the watch both fast and sharp. The work serves safety and community teams on live platforms. It spends big-model effort where it counts.
Catalog record
| CREATOR | Han Wang, Deyi Ji, Lanyun Zhu, Jiebo Luo, Roy Ka-Wei Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22738
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StreamSense: Streaming Social Task Detection with Selective Vision–Language Model Routing Conference: Proceedings of the ACM Web Conference 2026; April 13–17, 2026; Dubai, United Arab Emirates Proceedings of the ACM Web Conference 2026 (WWW ’26), April 13–17, 2026, Dubai, United Arab Emirates DOI: 10.1145/3774904.3793046 ISBN: 979-8-4007-2307-0/2026/04 ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Computer vision Han Wang, Deyi Ji, Lanyun Zhu, Jiebo Luo, Roy Ka-Wei Lee Abstract . Live streaming platforms require real-time monitoring and reaction to social signals, utilizing partial and asynchronous evidence from video, text, and audio. We propose StreamSense , a streaming detector that couples a lightweight streaming encoder with selective routing to a Vision–Language Model (VLM) expert. StreamSense handles most timestamps with the lightweight streaming encoder, escalates hard/ambiguous cases to the VLM, and defers decisions when context is insufficient. The encoder is trained using (i) a cross-modal contrastive term to align visual/audio cues with textual signals, and (ii) an IoU-weighted loss that down-weights poorly overlapping target segments, mitigating label interference across segment boundaries. We evaluate StreamSense on multiple social streaming detection tasks (e.g., sentiment classification and hate content moderation), and the results show that StreamSense achieves higher accuracy than VLM-only streaming while only occasionally invoking the VLM, thereby reducing average latency and compute. Our results indicate that selective escalation and deferral are effective primitives for understanding streaming social tasks. Code is publicly available on GitHub . † † cc-license: by 1. Introduction Figure 1. Comparison of (1) offline versus streaming video understanding and (2) general versus social task detection. Neg = Negative, Hate = Hate Speech. Live streaming platforms (e.g., Twitch, YouTube Live, TikTok Live, etc.) deliver low-latency video to millions while enabling real-time interaction, creating a tight feedback loop between content creators and their audiences. These platforms are popular for gaming, entertainment, live shopping, education, breaking news, and civic events, and underpin the creator economy. In this setting, understanding social signals as they emerge is crucial: streams exhibit rapid sentiment shifts, harassment waves, hate raids, coordinated misinformation during evolving events, deceptive promotions, and context-dependent toxicity. These signals are multimodal (spoken/on-screen text often provides decisive cues while visual and audio signals provide context) and asynchronous, complicating instant interpretation. Effective moderation, safety interventions, and analytics, therefore, require real-time inference under partial context; deciding when current evidence suffices, when to wait for more, and when to escalate to stronger models. Therefore, we formalize streaming social task detection (SSTD) as the problem of making a decision at each timestamp using past-only evidence from video, text, and audio. As illustrated in Figure 1 , SSTD differs from conventional (offline/general) video detection along three axes: (i) Causality & latency : predictions cannot peek into future content and must be synchronized with the live stream; (ii) Multimodality : while general task detection is often vision-centric, social tasks frequently hinge on spoken or on-screen language with visuals providing situational context; and (iii) Social inference : detection requires substantial human understanding and background knowledge to interpret intent, stance, or toxicity beyond visually localized cues. These differences make offline or general task methods unfit for streaming social analysis. Prior work on offline social tasks typically couples frozen encoders with deep classifiers ( 44 ; 20 ; 36 ; 15 ; 40 ; 28 ) . With the rise of Large Language Models (LLMs) and Vision-Language Models (VLMs), recent studies show strong performance on social nuanced tasks due to broad pretraining on social media and web content ( 46 ; 14 ; 19 ; 22 ; 24 ) . However, directly applying LLM/VLM systems to SSTD faces two obstacles: (1) integration : bringing new modality streams into large models typically requires costly multimodal pretraining, and (2) efficiency : high-capacity models are computationally expensive and slow for continuous, large-scale, real-time inference. Streaming work in general task detection is mainly focused on online action recognition (OAR) ( 7 ; 35 ; 38 ; 13 ; 37 ) . Those labels are time-synchronous: When the action occurs, the ground truth is immediately knowable. In contrast, for social tasks, annotators often cannot commit at a given timestamp without observing full context. Proper stream-level supervision should therefore allow an Unknown state until sufficient evidence accrues (Figure 2 ). In practice, available datasets only provide segment-level labels created after viewing the whole segment, which induces two pathologies for streaming models: (a) label interference between adjacent windows with conflicting segment labels in training-time, and (b) forced predictions under insufficient context. Unsurprisingly, naively adapting OAR architectures to SSTD yields weak results ( 34 ) . To address these research gaps, we propose StreamSense , a real-time multimodal system that combines a lightweight streaming encoder with selective routing to a high-capacity VLM expert, and a learned deferral policy when context is insufficient. The streaming encoder ingests past-only video, text and audio; a router estimates probability score and either (i) emits a prediction, (ii) escalates the current window to the VLM, or (iii) defers until more evidence arrives. To train robustly under segment-level supervision, we use (i) a cross-modal contrastive objective to align visual/audio cues with decisive textual signals, and (ii) an IoU-weighted cross-entropy that down-weights windows with low overlap to the target segment, mitigating label interference across adjacent windows. This design outperforms VLM-only baselines while sparingly invoking the VLM, yielding lower latency and compute. We summarize our contributions as follows: (1) We formalize streaming social task detection and articulate why it fundamentally differs from conventional video detection in terms of causality, multimodality, and social inference; (2) We introduce StreamSense , a selective-routing framework that combines a lightweight streaming encoder with an on-demand VLM expert and a deferral policy for low-context timestamps; and (3) We address supervision mismatch with IoU-weighted classification to reduce label interference and a cross-modal contrastive alignment to enhance modality fusion, achieving strong accuracy while substantially reducing VLM calls. 2. Related Works 2.1. Social Task Detection Social task detection studies human behaviors and intentions in social media content across modalities (text, images/memes, and video). We focus on video settings with representative subtasks, including sentiment/affect classification, hate/harassment detection, and misinformation identification. Sentiment is the most resourced area, with IEMOCAP ( 3 ) , CMU-MOSI ( 42 ) , CMU-MOSEI ( 41 ) , MELD ( 25 ) , CH-SIMS ( 39 ) , and CMU-MOSEAS ( 43 ) offering video- or segment-level annotations (sometimes fine-grained emotions). MOSI/MOSEI are particularly popular due to their strict guidelines, which ensure each segment conveys a complete and consistent opinion. Hate/harassment datasets such as HateMM ( 6 ) , MultiHateClip ( 30 ) , HateClipSeg ( 34 ) , and ImpliHateVid ( 26 ) provide video/segment labels and, in some cases, victim categories. However, outside HateClipSeg, segment boundaries are subjective and may fail to capture complete or consistent opinions. Misinformation video resources remain scarce ( 23 ; 2 ) and typically lack frame/segment alignment. Finally, these segment-level corpora are suited for offline analysis but not for stream-level supervision (i.e., lacking an Unknown state). Traditional social video detection and understanding models fuse unimodal features using deep architectures ( 44 ; 20 ; 36 ; 18 ; 15 ; 40 ; 28 ; 29 ) . Canonical examples include TFN ( 44 ) (tensor fusion of uni/bi/trimodal interactions), MISA ( 15 ) (modality-invariant/specific factors),