VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding

Junbo Zou, Ziheng Huang, Shengjie Zhang, Liwen Zhang, Weining Shen · 2026

Read VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding by Junbo Zou, Ziheng Huang, Shengjie Zhang, Liwen Zhang, Weining Shen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04094.txt.

Description

Long videos overflow a vision model's memory, so most systems sample frames blindly. Junbo Zou and 4 co-authors built VideoBrain, which learns where to look. Their 2026 framework sends two agents over the video: one retrieves frames by semantic match, the other samples densely inside promising intervals.

The vision-language model itself judges whether it has seen enough, rather than obeying a text-only controller. A behavior-aware reward stops the model from calling agents wastefully. Training data teaches when each agent helps. Adaptive sampling with recovery from bad choices beats both uniform sampling and one-pass keyframe selection on long-form understanding.

Catalog record

CREATORJunbo Zou, Ziheng Huang, Shengjie Zhang, Liwen Zhang, Weining Shen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04094

Browse the full library · About the Tower of Babel