MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan · 2026
Read MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding by Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-22932.txt.
Description
Wenhui Tan and seven co-authors teach video models to pick their own key frames. Long-form video understanding strains multimodal models, and the team's bet is that only a small frame subset truly matters per question. MSJoE, MLLM-Sampler Joint Evolution, evolves the model and a lightweight sampler together.
The system first reasons out queries covering diverse visual angles on the question. Those queries meet a frozen CLIP model to form a query-frame similarity matrix, from which the sampler predicts weights and selects a compact informative frame set for answer generation. Joint reinforcement learning co-adapts query reasoning, frame sampling, and frame understanding. A new dataset of about 2,800 videos with 7,000 question-answer pairs supports training, and tests show eight percent gain over the base model, beating the strongest baseline by about one percent.
Catalog record
| CREATOR | Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian Luan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-22932