StreamChat: Chatting with Streaming Video
Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, Jose M. Alvarez · 2024
Read StreamChat: Chatting with Streaming Video by Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, Jose M. Alvarez (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08646.txt.
Description
Jihao Liu and co-authors wrote this machine learning paper, published in two thousand twenty-four, on chatting with live video streams. A viewer may ask a question mid-stream, but most models answer from the single moment the question lands, missing what happens next.
StreamChat refreshes the model's view of the video at every step while it writes its reply. A cross-attention design keeps that constant refreshing cheap enough for live use. The team also built a dense instruction data set to train such streaming chat, and a coding scheme that stamps the relative time of words and frames. On standard image and video tests the model held its own with top systems. On live-stream tasks it beat them clearly. The authors aim at assistants that can follow events as they unfold.
Catalog record
| CREATOR | Jihao Liu, Zhiding Yu, Shiyi Lan, Shihao Wang, Rongyao Fang, Jan Kautz, Hongsheng Li, Jose M. Alvarez |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08646