Make Your Training Flexible: Towards Deployment-Efficient Video Models

Chenting Wang, Kunchang Li, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang · 2025

Read Make Your Training Flexible: Towards Deployment-Efficient Video Models by Chenting Wang, Kunchang Li, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14237.txt.

Introduction

Chenting Wang and five co-authors free video models from a fixed grid of frame tokens. Stock training takes a set token count off a stiff grid, which wastes work on dull spans and locks the model to one cost grade. Their 2025 tool, Flux, bends the grid and picks the best tokens a span has to give. A new test rule, token thrift, grades models on how much sense they pull from a capped token count.

Flux drops into most stock video frames at near no extra cost. Set in big pre-training, the FluxViT model took new top marks on a wide task set at stock cost. With just a fourth of the tokens, it still tied past top models under the thrift rule, a saving near 90 percent. Code and data sit open.

Catalog record

CREATORChenting Wang, Kunchang Li, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Make Your Training Flexible: Towards Deployment-Efficient Video Models Chenting Wang, Kunchang Li, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin Wang Abstract Popular video training methods mainly operate on a fixed number of tokens sampled from a predetermined spatiotemporal grid, resulting in sub-optimal accuracy-computation trade-offs due to inherent video redundancy. They also lack adaptability to varying computational budgets for downstream tasks, hindering applications of the most competitive model in real-world scenes. We thus propose a new test setting, Token Optimization, for maximized input information across budgets, which optimizes the size-limited set of input tokens through token selection from more suitably sampled videos. To this end, we propose a novel augmentation tool termed Flux. By making the sampling grid flexible and leveraging token selection, it is easily adopted in most popular video training frameworks, boosting model robustness with nearly no additional cost. We integrate Flux in large-scale video pre-training, and the resulting FluxViT establishes new state-of-the-art results across extensive tasks at standard costs. Notably, with 1/4 tokens only, it can still match the performance of previous state-of-the-art models with Token Optimization, yielding nearly 90% savings. All models and data are available at https://github.com/OpenGVLab/FluxViT . 1 Introduction Figure 1 : Flux (right) employs flexible sampling and token selection to achieve Token Optimization. Common methods(left) use rigid sampling(and use token reduction for applications directly). † † \heartsuit Corresponding authors. Video representation learning [ 81 , 50 , 22 ] is a fundamental task in computer vision, as it is critical in developing spatiotemporal perception and procedural reasoning in multimodal large language models [ 46 , 62 , 43 , 87 ] and embodied AI systems [ 27 , 88 ] . However, the deployment of video models is computationally intensive due to the high volume of video tokens compared to image tokens. Popular video training methods work on fixed-count patched tokens sampled from a predefined spatiotemporal grid (e.g., 8 \times 224 2 ), leading to considerable redundancy in training and deployment, especially for long, high-resolution videos. Moreover, this inflexible and fixed training achieves sub-optimal results for tasks that require dynamic resolution settings, including restricted computation or emphasizing frame counts rather than frame resolution. Researchers have developed different approaches for better trade-offs. First, to fit a limited computation budget in inference, many works [ 16 , 67 , 35 ] use token reduction on the same densely sampled tokens as in training, showing sub-optimal performance. For one thing, there is a trade-off between the complexity (cost) and performance of the token reduction strategy. Current strategies can not perform well with a significant reduction rate. For another, the inflexibly trained models can not generalize with the sparse, masked tokens. Second, existing methods for flexible network training, such as Resformer [ 75 ] and FFN [ 98 ] , have demonstrated the efficacy of flexible networks operating at different spatial or temporal resolutions but not both simultaneously. Although these approaches achieve robust performance across diverse settings, we contend that merely reducing frame counts or resolution under computational constraints fails to optimize token capacity utilization and remains susceptible to redundancy. Notably, FFN exhibits substantial performance degradation when operating with only two frames in their research. We argue that the input token set with a size equal to 2 frames can be optimized. Also, they are not validated in large-scale pre-training to achieve competitive results for real-world applications, and the performance gain with larger resolution input comes with quadratic computation cost increase, which is also insufficient. We propose a new perspective of the desired optimal computation-and-accuracy and spatial-and-temporal trade-offs across various settings, termed ‘Token Optimization’ (TO in short), as shown in Figure 1 . It involves selecting an optimized set of input tokens based on budget from more properly sampled videos for information maximization. Since there is no perfect token selector for all settings, we promote a better solution of using flexible sampling, with more densely and finely sampled videos for higher computation and more sparsely sampled videos for lower budget. Spatial and temporal trade-offs are also achieved. To better achieve TO, we propose a new data augmentation tool for training flexible video models, termed Flux for Fl exibility in u niversal conte x ts, by utilizing the combination of flexible sampling and token selector to make training flexible while not introducing costs. We integrate it with the Unmasked Teacher [ 47 ] pre-training, supervised-training, multi-modal contrastive training, and chat-centric training to validate its efficiency. Our new recipe brings more robustness caused by Flux augmentation both in standard settings and our TO setting. Two plug-in modules are used for better feature extraction regarding the reduced token set with flexible reduction rates and input spatiotemporal resolutions. We build bottom-up ablation studies in turning the model flexible. The resulting model FluxViT achieves supreme improvements regarding the currently state-of-the-art InternVideo2-Dist series models in fair settings across budgets and across tasks, as in Figure 2 . Figure 2 : Overview of our Flux method. The same-scaled FluxViT and InternVideo2 [ 87 ] series models are both pre-trained with the InternVideo2-1b model as the teacher using the same dataset. The “FluxViT+” refers to the results using Token Optimization at test time with the same GFLOPS. Our FluxViT-S outperforms the previous state-of-the-art small-scale model, InternVideo2-S [ 87 ] , by 2.2% on K400 under standard computation constraints. Moreover, it achieves comparable performance to InternVideo2-S while utilizing only about 10% of inference cost. FluxViT-B achieves competitive results with state-of-the-art much larger scale models on various tasks, including scene-based action recognition task [ 38 ] ( 90.0% top-1 accuracy on K400), motion-intensive [ 26 ] ( 75.8% top-1 accuracy on SSv2), long-term task [ 74 ] ( 94.1% top-1 accuracy on COIN), zero-shot text-to-video retrieval [ 91 , 1 , 31 , 69 , 13 ] ( 49.9% R@1 on MSRVTT, 53.5% R@1 on DiDeMo and 56.7% R@1 on ActivityNet). Notably, it maintains state-of-the-art performance among the similar scaled models while using only 1/4 to 1/2 of their tokens, resulting in 70% to 95% savings in computation. On chat centric tasks including MVbench [ 46 ] for general perception and Dream-1k [ 80 ] for fine-grained caption, our model surpasses widely used SigLIP, CLIP and UMT in ‘linear prob’ setting across computation budgets, where only the projector between the ViT and LLM is trained for comparability regarding the ViT’s capability as the common stage-1 training in chat models. This highlights Flux’s strong effect across computation constraints and tasks and underscores the robustness afforded by such flexibility. 2 Related Work Flexible Model Training. Many recent studies focus on developing a method to train a single model that performs well across various settings, facilitating seamless adaptation to diverse downstream deployment scenarios. These methods include training a model as a set of shared weights to be pruned for specific tasks [ 94 , 8 ] , as well as optimizing models for varied spatial resolutions [ 75 ] , frame counts [ 98 ] , spatiotemporal resolutions [ 57 , 83 , 54 , 90 ] , patch sizes [ 6 ] , and representation dimensions [ 41 , 9 ] . The Resformer [ 75 ] and FFN [ 98 ] methods propose co-training with three different resolution settings

Read the complete text