Pretrained Image-Text Models are Secretly Video Captioners

Chunhui Zhang, Yiren JianZhongyu OuyangSoroush VosoughiDepartment of Computer Science, Dartmouth College {chunhui.zhang.gr, yiren.jian.gr, zhongyu.ouyang.gr, soroush.vosoughi}@dartmouth.edu · 2025

Read Pretrained Image-Text Models are Secretly Video Captioners by Chunhui Zhang, Yiren JianZhongyu OuyangSoroush VosoughiDepartment of Computer Science, Dartmouth College {chunhui.zhang.gr, yiren.jian.gr, zhongyu.ouyang.gr, soroush.vosoughi}@dartmouth.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13363.txt.

Introduction

Chunhui Zhang, Yiren Jian, Zhongyu Ouyang, and Soroush Vosoughi showed in 2025 that an image caption model can be taught to caption videos on the cheap. Video caption tools cost a great deal to build, and motion makes the task hard. The team took a known image model and fine-tuned it on just six thousand video-text pairs, far fewer than rival tools use. Frames were simply strung side by side as input.

The adapted model placed near the top on three known test sets. It beat many tools built just for video. The study also weighed model size, data thrift, and a reward-based tune-up. The authors pitch the light route as a sound choice where compute is scarce.

Catalog record

CREATORChunhui Zhang, Yiren JianZhongyu OuyangSoroush VosoughiDepartment of Computer Science, Dartmouth College {chunhui.zhang.gr, yiren.jian.gr, zhongyu.ouyang.gr, soroush.vosoughi}@dartmouth.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13363

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Pretrained Image-Text Models are Secretly Video Captioners Chunhui Zhang, Yiren JianZhongyu OuyangSoroush VosoughiDepartment of Computer Science, Dartmouth College {chunhui.zhang.gr, yiren.jian.gr, zhongyu.ouyang.gr, soroush.vosoughi}@dartmouth.edu Abstract Developing video captioning models is computationally expensive. The dynamic nature of video also complicates the design of multimodal models that can effectively caption these sequences. However, we find that by using minimal computational resources and without complex modifications to address video dynamics, an image-based model can be repurposed to outperform several specialised video captioning systems. Our adapted model demonstrates top-tier performance on major benchmarks, ranking 2nd on MSR-VTT and MSVD, and 3rd on VATEX. We transform it into a competitive video captioner by post-training a typical image captioning model BLIP-2 with only 6,000 video-text pairs and simply concatenating frames—significantly fewer data than other methods, which use 2.5 to 144 million pairs. From a resource optimization perspective, this video captioning study focuses on three fundamental factors: optimizing model scale, maximizing data efficiency, and incorporating reinforcement learning. This extensive study demonstrates that a lightweight, image-based adaptation strategy can rival state-of-the-art video captioning systems, offering a practical solution for low-resource scenarios. † † ∗ Equal contribution and random order. 1 Introduction Vision-language pretraining significantly advances multimodal tasks such as captioning, question answering, retrieval and broader video understanding ( Liu et al. 2023b ; Liu et al. 2023a ; Li et al. 2023b ; Dai et al. 2023a ; Chen et al. 2023b ; Kuo et al. 2023 ; Xu et al. 2023 ; Diao et al. 2023 ; Diao et al. 2024 ; Diao et al. 2025 ; Zhang et al. 2022a ; Liu et al. 2024 ; Han et al. 2024 ; Jian et al. 2023 ; Jian et al. 2024 ) . Among these, video captioning stands out as it narrates visual concepts and their temporal interactions, reflecting the intricate multimodal processes as humans to perceive and articulate dynamic visual experiences. Current video-text methods often incorporate intricate designs tailored to video inputs. For instance, some models extend existing frameworks by integrating frame samplers to capture temporal dynamics ( Alayrac et al. 2022 ; Yang et al. 2021 ; Xu et al. 2021 ) . Other approaches, such as ALPRO ( Li et al. 2022a ) and VIOLET ( Fu et al. 2023 ) , propose end-to-end models that are meticulously trained on large-scale video-text datasets sourced from the Web ( Zellers et al. 2021 ; Bain et al. 2021 ) . Despite their success, video captioning models remain highly resource-intensive, often hitting performance bottlenecks when (i) computational resources are constrained, or (ii) the task requires specialized priors without clear guidance for model design and training. This raises a critical question: for simplicity and efficiency, how can we repurpose existing image captioning models for video captioning, without relying on complex, hand-crafted video-specific designs? Figure 1: Key factors in recycling BLIP for video captioning: Model – assessing the scale and trainability of components like the ViT, LLM, and Q-Former; Data – examining the volume, quality, and fusion strategies for image and video-text pairs; Supervision – employing reinforcement learning to align generated captions with human language quality standards (CIDEr). To address this, we revisit fundamental factors in training— model scale , data efficiency , and supervision —that critically influence video captioning while being agnostic to the variants of video-specific designs: First , we find that moderate-sized language models (LMs) when fine-tuned for specific tasks, can meet the demands of video captioning efficiently. This challenges the common belief that larger models are always superior, demonstrating that targeted optimization can outperform sheer model size. Second , using extensive pretraining on image-text pairs, as demonstrated with BLIP-2, is transferable to video tasks. This allows the model to achieve high performance with minimal video usage, offering an efficient alternative to training from scratch. Third , instead of relying on traditional cross-entropy loss, we optimize directly for non-differentiable CIDEr with reinforcement learning, ensuring that the generated captions better align with human-standard video descriptions. By bypassing complex, specialized video input designs, our experiments demonstrate that BLIP-2 straightforwardly derived from image captioning, can be effectively optimized to deliver competitive video captioning performance. This study underscores the potential of simplicity and efficiency in advancing multimodal video captioning, providing a streamlined yet stable solution. The codes are released: https://github.com/chunhuizng/mllm-video-captioner . 2 Recycling BLIP-2 for Video Captioning As shown in Fig. 1 , we adapt BLIP-2, a typical image-text model (details in App. B ), for video captioning without any additional parameters. Each video frame is encoded by ViT, which generates visual tokens that are concatenated to form a unified representation (e.g., an 8-frame video produces a token sequence of size 8 \times 256). This unified token sequence is then processed by the Q-former and passed to the LM to generate captions. 3 Training Recipes: Model, Data, and Supervision According to Tab. 1 , our solution has top-level performance on important benchmarks (particularly on the CIDEr metric-the primary ranking measure on Paperswithcode ), ranking 2nd on MSR-VTT and MSVD, and 3rd on VATEX, among models with publicly available code. More importantly, it proves to be highly efficient without any video architecture design, using only 6k video-text pairs—significantly less than the million-level datasets required by competing baselines. Additional background is in App. A . The settings are detailed in App. C , and further experiments ( ablations, other datasets, and other video tasks ) supporting the following analysis are in App. D . Model MSR-VTT ( Xu et al. 2016 ) MSVD ( Chen and Dolan 2011 ) VATEX ( Wang et al. 2019 ) Code # msr v.- t. pairs C. M. R. B4. P. C. M. R. B4. P. C. M. R. B4. P. - IcoCap 60.2 31.1 64.9 47.0 - 110.3 39.5 76.5 59.1 - 67.8 25.7 53.1 37.4 - No - MaMMUT 73.6 - - - 77.5 195.6 - - - 85.6 - - - - 79.9 No - VideoCoCa 73.2 - 68.0 53.8 - - - - - - 77.8 - 54.5 39.7 - No 144.7M VALOR 74.0 32.9 68.0 54.4 81.0 178.5 51.0 87.9 80.7 83.7 95.8 29.4 57.4 45.6 73.3 Yes 1.18M VLAB 74.9 33.4 68.3 54.6 - 179.8 51.2 87.9 79.3 - - - - - - No 10.7M GIT2 75.9 33.1 68.2 54.8 75.4 - - - - - - - - - - Yes - VAST 78.0 - - 56.7 77.2 - - - - - 99.5 - - 45.0 81.9 Yes 27M mPLUG-2 80.0 34.9 70.1 57.8 82.7 165.8 48.4 85.3 70.5 82.5 - - - - - Yes 2.5M Ours 79.5 34.2 68.3 52.4 81.2 168.0 48.3 85.8 73.5 84.4 87.1 29.1 56.7 43.3 82.1 Yes 6K Table 1: Overall comparison. The results for MSR-VTT, MSVD, and VATEX are from the PaperswithCode open leaderboard. The abbreviations C., M., R., B4., and P. stand for CIDEr, METEOR, ROUGE-L, BLEU-4, and PAC-S Sarto et al. 2023 , respectively. We choose CIDEr as the most referential metric, following the PaperswithCode . Tab. 2 has details about configs and references. 3.1 Model Scale Trainability: modal connector > LLM > ViT Figure 2: Comparisons of different setups for models on the MSR-VTT dataset: (a) freezing modules, (b) scales of LLMs, (c) usage of image-text pairs

Read the complete text