Apollo: An Exploration of Video Understanding in Large Multimodal Models
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia · 2024
Read Apollo: An Exploration of Video Understanding in Large Multimodal Models by Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10360.txt.
Introduction
Zohar and co-authors dissect what makes large AI models good at understanding video, then build their own family on the lessons. Design choices in this field are often made blind, and training costs block careful study. Their 2024 work first shows that choices tuned on small models and datasets transfer to larger ones up to a point, which makes cheap experiments trustworthy.
Using that shortcut, the team tested frame sampling, vision encoders, data mixes, and training schedules. Sampling by frame rate beat even spacing, and some encoders clearly led for video. The resulting Apollo models read hour-long videos efficiently. The smallest beat most rival models twice its size on a long-video test, and the largest led its class on two benchmarks. The study doubles as a recipe book.
Catalog record
| CREATOR | Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10360
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Apollo: An Exploration of Video Understanding in Large Multimodal Models Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, Xide Xia Abstract Despite the rapid integration of video perception capabilities into Large Multimodal Models (LMMs), the underlying mechanisms driving their video understanding remain poorly understood. Consequently, many design decisions in this domain are made without proper justification or analysis. The high computational cost of training and evaluating such models, coupled with limited open research, hinders the development of video-LMMs. To address this, we present a comprehensive study that helps uncover what effectively drives video understanding in LMMs. We begin by critically examining the primary contributors to the high computational requirements associated with video-LMM research and discover Scaling Consistency , wherein design and training decisions made on smaller models and datasets (up to a critical size) effectively transfer to larger models. Leveraging these insights, we explored many video-specific aspects of video-LMMs, including video sampling, architectures, data composition, training schedules, and more. For example, we demonstrated that fps sampling during training is vastly preferable to uniform frame sampling and which vision encoders are the best for video representation. Guided by these findings, we introduce Apollo , a state-of-the-art family of LMMs that achieve superior performance across different model sizes. Our models can perceive hour-long videos efficiently, with Apollo- 3 3 B outperforming most existing 7 7 B models with an impressive 55.1 on LongVideoBench. Apollo- 7 7 B is state-of-the-art compared to 7B LMMs with a 70.9 70.9 on MLVU, and 63.3 63.3 on Video-MME. † † date: August 24, 2026 † † correspondence: Orr Zohar at [email protected] , Xiaohan Wang at [email protected] † † Project Page: https://apollo-lmms.github.io 1 Introduction Despite the rapid advancements in language and image-language modeling ( Hoffmann et al., 2022 ; Brown, 2020 ; Yang et al., 2024 ; Liu et al., 2024a ; Alayrac et al., 2022 ; Laurençon et al., 2024a ; OpenAI, 2024 ) , the development of video Large Multimodal Models (video-LMMs) has not kept pace. Videos provide a rich, dynamic information source, capturing nuanced temporal and spatial features beyond the reach of static images. However, video-LMMs remain under-explored, hampered by unique challenges: notably higher computational demands and a broader, more complex design space compared to their image-based counterparts ( Li et al., 2023a ; Li et al., 2025 ; Liu et al., 2024d ; Li et al., 2024b ; Xu et al., 2024a ) . Many fundamental questions about video-LMM design remain unanswered: How should videos be sampled? Which vision encoders yield optimal representations? What are the best practices for resampling video tokens? Early approaches primarily extended image-LMMs directly ( Xu et al., 2024b ; Kim et al., 2024 ; Wu, 2024 ; Zhang et al., 2024e ) or with video-specific fine-tuning ( Li et al., 2023a ; Zhang et al., 2023 ; Maaz et al., 2023 ) . Recent methods introduced diverse design choices, such as longer context windows ( Zhang et al., 2024e ) , multi-modality mixing ( Li et al., 2024a ; Li et al., 2024c ) , agent workflows ( Wang et al., 2024c ) , self-training ( Zohar et al., 2024 ) , and more. Despite these efforts, the impact of these design decisions on video-LMM performance is poorly understood. This lack of systematic investigation motivates our study. To overcome the computational challenges of training video-LMMs, we explore whether design decisions from smaller models correlate effectively with larger ones. Traditional scaling laws ( Hoffmann et al., 2022 ) predict model performance based on size, but apply to models trained from scratch and require training multiple models to predict performance. Scaling laws have also been observed in LMM pretraining ( Aghajanyan et al., 2023 ; Yu et al., 2023 ) . Since LMMs integrate multiple pre-trained components, it’s uncertain if these laws hold. By relaxing scaling laws, our experiments reveal that design choices made with smaller LMMs transfer to larger ones, a phenomenon we term Scaling Consistency (Sec. 3 ). Utilizing these insights, we conduct an extensive study across the video-LMM design space, addressing essential aspects of video-language modeling, such as video sampling and encoding methods, token resampling and integration strategies, and data compositions (Sec. 4 & 5 ). For instance, we discover that frames-per-second video sampling significantly outperforms standard uniform sampling used in previous works ( Liu et al., 2024d ; Liu et al., 2024b ) . We also find which vision encoder combinations are the most robust and that the Perceiver Resampler ( Jaegle et al., 2021 ) outperforms average pooling. When studying the numerous benchmarks available, we discovered a large portion of the performance improvements are driven primarily via language modeling and, therefore, curate ApolloBench, which significantly reduces evaluation time while improving assessment quality (Sec. 2 ). Figure 1: Apollo exploration. Schematic illustrating our comprehensive exploration of video-specific design choices; critically evaluating the existing conceptions in the field, from video sampling and model architecture to training schedules and data compositions. For example, we found that the SigLIP encoder is the best single encoder for video-LMMs but can be combined with additional encoders to improve temporal perception, and that keeping a \sim 10\% text data during fine-tuning is critical for video understanding performance. More insights can be found in Sec. 4 & 5 . Building upon our findings, we introduce Apollo, a family of state-of-the-art LMMs capable of comprehending hour-long videos. Apollo models exhibit strong performance across various scales. Notably, Apollo- 3 B surpasses most existing 7 B models, achieving scores of 58.4~(+12.8) on Video-MME (w/o sub.), 68.7~(+6.9) on MLVU, and 62.7~(+14.1) on ApolloBench. Apollo- 7 B attains impressive scores of 61.2~(+0.6) on Video-MME (w/o sub.), 70.9~(+5.4) on MLVU, and 66.3~(+2.4) on ApolloBench, making it competitive with 30 B models. Our contributions are as follows: 1. We conduct a systematic exploration of the video modeling design space for Large Multimodal Models, uncovering critical factors that drive performance and providing actionable insights for future research. 2. We identify Scaling Consistency, where design decisions effective for smaller LMMs and datasets are transferred effectively to larger ones, reducing computational costs and enabling efficient experimentation. 3. We address evaluation inefficiencies by curating ApolloBench, a subset of existing benchmarks that cuts evaluation time by 41\times while offering detailed insights into temporal reasoning and perception tasks. 4. We introduce Apollo, a family of LMMs that achieves state-of-the-art results across video understanding multiple benchmarks. Notably, Apollo- 3 B surpasses nearly all 7 B models, while Apollo- 7 B variant is state-of-the-art among models with less than 30 B parameters. In Sec 2 , we analyze the state of video benchmarks and introduce ApolloBench. In Sec. 3 , we show how one can relax traditional scaling laws for computational savings. In Sec. 4 , we explore the architecture design space. In Sec. 5 , we investigate different training protocols and data mixtures. Finally, in Sec. 6 , we present Apollo, a state-of-the-art family of video-LMMs. Figure 2: Benchmark Analysis. (Left) Accuracy of the open-source LMMs on various video question-answering benchmarks when provided with different input modalities: full video (green bars), a single frame from the video (red bars), and text-only input without