MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding

Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan · 2026

Read MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding by Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-22932.txt.

Description

Wenhui Tan and seven co-authors teach video models to pick their own key frames. Long-form video understanding strains multimodal models, and the team's bet is that only a small frame subset truly matters per question. MSJoE, MLLM-Sampler Joint Evolution, evolves the model and a lightweight sampler together.

The system first reasons out queries covering diverse visual angles on the question. Those queries meet a frozen CLIP model to form a query-frame similarity matrix, from which the sampler predicts weights and selects a compact informative frame set for answer generation. Joint reinforcement learning co-adapts query reasoning, frame sampling, and frame understanding. A new dataset of about 2,800 videos with 7,000 question-answer pairs supports training, and tests show eight percent gain over the base model, beating the strongest baseline by about one percent.

Catalog record

CREATORWenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-22932

Browse the full library · About the Tower of Babel